Skip to contentSkip to conversation

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

DISCUSSION Emmanuel Lacour 4 participants 8 messages Nov 28, 2013 – Dec 11, 2013 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 11 lines · Emmanuel Lacour

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 10 lines · Dave Chinner
Quoted 9 lines · Emmanuel Lacour

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

Quoted 8 lines
Quoted 2 lines

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:

http://people.easter-eggs.org/~manu/xfs.log

This may be related to a friend problem here:

http://tracker.ceph.com/issues/6386

thanks for any help on solving this!

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:

Quoted 8 lines
Quoted 2 lines

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:

http://people.easter-eggs.org/~manu/xfs.log

OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.

Yup, there's your problem:

[4583991.478469] ceph-osd        D ffff88047fc93f40     0 22951
1 0x00000004
[4583991.478471]  ffff88046d241140 0000000000000082 ffffffff81047e75
ffff88046f949800
[4583991.478475]  0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8
ffff88046d241140
[4583991.478479]  0000000000000000 00000001444d68bd ffff88046d241140
0000000000000005
[4583991.478483] Call Trace:
[4583991.478487]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.478491]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.478494]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.478498]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.478502]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.478505]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.478518]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.478535]  [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs]
[4583991.478550]  [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs]
[4583991.478566]  [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs]
[4583991.478581]  [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs]
[4583991.478594]  [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs]
[4583991.478606]  [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs]
[4583991.478610]  [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f
[4583991.478614]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.478618]  [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257
[4583991.478622]  [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187
[4583991.478625]  [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce
[4583991.478629]  [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68
[4583991.478633]  [<ffffffff81395429>] ?  system_call_fastpath+0x16/0x1b

It'll be stuck on this:

	hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);

which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.

Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!

I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.

Quoted 3 lines

This may be related to a friend problem here:

http://tracker.ceph.com/issues/6386

Doesn't look related, unless the OOM killer is being triggered somehow...

Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:

[4583991.476833] ceph-osd        D ffff88047fc33f40     0 11072 1 0x00000004
[4583991.476836]  ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040
[4583991.476840]  0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040
[4583991.476844]  0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005
[4583991.476848] Call Trace:
[4583991.476852]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.476855]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.476859]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.476862]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.476867]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.476870]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.476883]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.476899]  [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs]
[4583991.476904]  [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30
[4583991.476909]  [<ffffffff81041942>] ? current_fs_time+0x27/0x2d
[4583991.476925]  [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs]
[4583991.476939]  [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs]
[4583991.476943]  [<ffffffff81114574>] ? lookup_fast+0x3d/0x215
[4583991.476954]  [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs]
[4583991.476966]  [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs]
[4583991.476970]  [<ffffffff81115fda>] ? vfs_create+0x62/0x8b
[4583991.476974]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.476978]  [<ffffffff81117179>] ? do_last+0x595/0xa16
[4583991.476982]  [<ffffffff811176be>] ? path_openat+0xc4/0x335
[4583991.476985]  [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e
[4583991.476989]  [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1
[4583991.476993]  [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0
[4583991.476996]  [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b

That one isn't so easy to fix, unfortunately.

Cheers,

Dave.

Signature · 19 lines
Dave Chinner
[email protected]

xfs: xfs_dir2_block_to_sf temp buffer allocation fails 

From: Dave Chinner <[email protected]>

If we are using a large directory block size, and memory becomes
fragmented, we can get memory allocation failures trying to
kmem_alloc(64k) for a temporary buffer. However, there is not need
for a directory buffer sized allocation, as the end result ends up
in the inode literal area. This is, at most, slightly less than 2k
of space, and hence we don't need an allocation larger than that
fora temporary buffer.

Signed-off-by: Dave Chinner <[email protected]>
---
 fs/xfs/xfs_dir2_sf.c | 58 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 24 deletions(-)
fs/xfs/xfs_dir2_sf.c +34 -24
index aafc6e4..3725fb1 100644
@@ -170,6 +170,7 @@ xfs_dir2_block_to_sf(
 	char			*ptr;		/* current data pointer */
 	xfs_dir2_sf_entry_t	*sfep;		/* shortform entry */
 	xfs_dir2_sf_hdr_t	*sfp;		/* shortform directory header */
+	xfs_dir2_sf_hdr_t	*dst;		/* temporary data buffer */
 
 	trace_xfs_dir2_block_to_sf(args);
 
@@ -177,35 +178,20 @@ xfs_dir2_block_to_sf(
 	mp = dp->i_mount;
 
 	/*
-	 * Make a copy of the block data, so we can shrink the inode
-	 * and add local data.
+	 * allocate a temporary destination buffer the size of the inode
+	 * to format the data into. Once we have formatted the data, we
+	 * can free the block and copy the formatted data into the inode literal
+	 * area.
 	 */
-	hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);
-	memcpy(hdr, bp->b_addr, mp->m_dirblksize);
-	logflags = XFS_ILOG_CORE;
-	if ((error = xfs_dir2_shrink_inode(args, mp->m_dirdatablk, bp))) {
-		ASSERT(error != ENOSPC);
-		goto out;
-	}
+	dst = kmem_alloc(mp->m_sb.sb_inodesize, KM_SLEEP);
+	hdr = bp->b_addr;
 
 	/*
-	 * The buffer is now unconditionally gone, whether
-	 * xfs_dir2_shrink_inode worked or not.
-	 *
-	 * Convert the inode to local format.
-	 */
-	dp->i_df.if_flags &= ~XFS_IFEXTENTS;
-	dp->i_df.if_flags |= XFS_IFINLINE;
-	dp->i_d.di_format = XFS_DINODE_FMT_LOCAL;
-	ASSERT(dp->i_df.if_bytes == 0);
-	xfs_idata_realloc(dp, size, XFS_DATA_FORK);
-	logflags |= XFS_ILOG_DDATA;
-	/*
 	 * Copy the header into the newly allocate local space.
 	 */
-	sfp = (xfs_dir2_sf_hdr_t *)dp->i_df.if_u1.if_data;
+	sfp = (xfs_dir2_sf_hdr_t *)dst;
 	memcpy(sfp, sfhp, xfs_dir2_sf_hdr_size(sfhp->i8count));
-	dp->i_d.di_size = size;
+
 	/*
 	 * Set up to loop over the block's entries.
 	 */
@@ -258,10 +244,34 @@ xfs_dir2_block_to_sf(
 		ptr += dp->d_ops->data_entsize(dep->namelen);
 	}
 	ASSERT((char *)sfep - (char *)sfp == size);
+
+	/* now we are done with the block, we can shrink the inode */
+	logflags = XFS_ILOG_CORE;
+	error = xfs_dir2_shrink_inode(args, mp->m_dirdatablk, bp);
+	if (error) {
+		ASSERT(error != ENOSPC);
+		goto out;
+	}
+
+	/*
+	 * The buffer is now unconditionally gone, whether
+	 * xfs_dir2_shrink_inode worked or not.
+	 *
+	 * Convert the inode to local format and copy the data in.
+	 */
+	dp->i_df.if_flags &= ~XFS_IFEXTENTS;
+	dp->i_df.if_flags |= XFS_IFINLINE;
+	dp->i_d.di_format = XFS_DINODE_FMT_LOCAL;
+	ASSERT(dp->i_df.if_bytes == 0);
+	xfs_idata_realloc(dp, size, XFS_DATA_FORK);
+
+	logflags |= XFS_ILOG_DDATA;
+	memcpy(dp->i_df.if_u1.if_data, dst, size);
+	dp->i_d.di_size = size;
 	xfs_dir2_sf_check(args);
 out:
 	xfs_trans_log_inode(args->trans, dp, logflags);
-	kmem_free(hdr);
+	kmem_free(dst);
 	return error;
 }
 

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 2 lines · Dave Chinner

On Tue, Dec 03, 2013 at 10:53:58AM +0100, Emmanuel Lacour wrote:
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size. Yup, there's your problem:

I can confirm I also saw this from times to times, in the very same context (ceph OSD, xfs volume with 64k directory blocks)

I never took the time to fully report the problem because it was very sporadic, and I suspected a specific option in my hand-made kernels, in which the problem seems to occurs more often. With the 'standard' debian 3.10.2 (from testing) I never saw the problem (56 days uptime)

cf http://tracker.ceph.com/issues/6301

the report isn't complete here ... sometimes I just have the deadlock, but without oops. can try to dig deeper inmy old logs to be sure it's really the same

Cheers,

Signature · 7 lines
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : [email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
In reply to Dave Chinner
Quoted 1 line · Dave Chinner

thanks very much for your quick and detailled answer!

Quoted 4 lines

OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.

yes, the 64k was taken from a too quickly read of advice, I don't think it's of any help on a ceph cluster but I'm not an FS guru. Is there a way to lower it at runtime?

Quoted 2 lines

Yup, there's your problem:

[...]

Quoted 4 lines

Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!

unfortunatly I cannot test this patch because:

- it's a production cluster and it's currently hard for me to reboot nodes (not enough nodes ;))
- just after hiting this problem I saw a kernel 3.11 available on Debian backports and decided to upgrade the whole cluster.

since this upgrade, there is no problems anymore ... I cross my fingers ;)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#6 lore raw
In reply to Dave Chinner
Quoted 121 lines · Dave Chinner
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:

Quoted 8 lines
Quoted 2 lines

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:

http://people.easter-eggs.org/~manu/xfs.log

OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.

Yup, there's your problem:

[4583991.478469] ceph-osd        D ffff88047fc93f40     0 22951
1 0x00000004
[4583991.478471]  ffff88046d241140 0000000000000082 ffffffff81047e75
ffff88046f949800
[4583991.478475]  0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8
ffff88046d241140
[4583991.478479]  0000000000000000 00000001444d68bd ffff88046d241140
0000000000000005
[4583991.478483] Call Trace:
[4583991.478487]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.478491]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.478494]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.478498]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.478502]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.478505]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.478518]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.478535]  [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs]
[4583991.478550]  [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs]
[4583991.478566]  [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs]
[4583991.478581]  [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs]
[4583991.478594]  [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs]
[4583991.478606]  [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs]
[4583991.478610]  [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f
[4583991.478614]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.478618]  [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257
[4583991.478622]  [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187
[4583991.478625]  [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce
[4583991.478629]  [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68
[4583991.478633]  [<ffffffff81395429>] ?  system_call_fastpath+0x16/0x1b

It'll be stuck on this:

	hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);

which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.

Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!

I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.

Quoted 3 lines

This may be related to a friend problem here:

http://tracker.ceph.com/issues/6386

Doesn't look related, unless the OOM killer is being triggered somehow...

Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:

[4583991.476833] ceph-osd        D ffff88047fc33f40     0 11072 1 0x00000004
[4583991.476836]  ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040
[4583991.476840]  0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040
[4583991.476844]  0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005
[4583991.476848] Call Trace:
[4583991.476852]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.476855]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.476859]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.476862]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.476867]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.476870]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.476883]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.476899]  [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs]
[4583991.476904]  [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30
[4583991.476909]  [<ffffffff81041942>] ? current_fs_time+0x27/0x2d
[4583991.476925]  [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs]
[4583991.476939]  [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs]
[4583991.476943]  [<ffffffff81114574>] ? lookup_fast+0x3d/0x215
[4583991.476954]  [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs]
[4583991.476966]  [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs]
[4583991.476970]  [<ffffffff81115fda>] ? vfs_create+0x62/0x8b
[4583991.476974]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.476978]  [<ffffffff81117179>] ? do_last+0x595/0xa16
[4583991.476982]  [<ffffffff811176be>] ? path_openat+0xc4/0x335
[4583991.476985]  [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e
[4583991.476989]  [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1
[4583991.476993]  [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0
[4583991.476996]  [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b
Quoted 22 lines

That one isn't so easy to fix, unfortunately.

Cheers,

Dave.

Signature · 16 lines
Dave Chinner
[email protected]

xfs: xfs_dir2_block_to_sf temp buffer allocation fails 

From: Dave Chinner <[email protected]>

If we are using a large directory block size, and memory becomes
fragmented, we can get memory allocation failures trying to
kmem_alloc(64k) for a temporary buffer. However, there is not need
for a directory buffer sized allocation, as the end result ends up
in the inode literal area. This is, at most, slightly less than 2k
of space, and hence we don't need an allocation larger than that
fora temporary buffer.

Signed-off-by: Dave Chinner <[email protected]>

D'oh, I missed this one too. If you stick 'patch' in the subject they'll have additional visibility.

Looks good to me.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 143 lines · Ben Myers
Quoted 142 lines · Dave Chinner
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour

Dear XFS users,

I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:

XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?

It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:

Quoted 8 lines
Quoted 2 lines

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:

http://people.easter-eggs.org/~manu/xfs.log

OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.

Yup, there's your problem:

[4583991.478469] ceph-osd        D ffff88047fc93f40     0 22951
1 0x00000004
[4583991.478471]  ffff88046d241140 0000000000000082 ffffffff81047e75
ffff88046f949800
[4583991.478475]  0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8
ffff88046d241140
[4583991.478479]  0000000000000000 00000001444d68bd ffff88046d241140
0000000000000005
[4583991.478483] Call Trace:
[4583991.478487]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.478491]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.478494]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.478498]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.478502]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.478505]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.478518]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.478535]  [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs]
[4583991.478550]  [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs]
[4583991.478566]  [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs]
[4583991.478581]  [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs]
[4583991.478594]  [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs]
[4583991.478606]  [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs]
[4583991.478610]  [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f
[4583991.478614]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.478618]  [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257
[4583991.478622]  [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187
[4583991.478625]  [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce
[4583991.478629]  [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68
[4583991.478633]  [<ffffffff81395429>] ?  system_call_fastpath+0x16/0x1b

It'll be stuck on this:

	hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);

which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.

Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!

I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.

Quoted 3 lines

This may be related to a friend problem here:

http://tracker.ceph.com/issues/6386

Doesn't look related, unless the OOM killer is being triggered somehow...

Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:

[4583991.476833] ceph-osd        D ffff88047fc33f40     0 11072 1 0x00000004
[4583991.476836]  ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040
[4583991.476840]  0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040
[4583991.476844]  0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005
[4583991.476848] Call Trace:
[4583991.476852]  [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28
[4583991.476855]  [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123
[4583991.476859]  [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d
[4583991.476862]  [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86
[4583991.476867]  [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb
[4583991.476870]  [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79
[4583991.476883]  [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs]
[4583991.476899]  [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs]
[4583991.476904]  [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30
[4583991.476909]  [<ffffffff81041942>] ? current_fs_time+0x27/0x2d
[4583991.476925]  [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs]
[4583991.476939]  [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs]
[4583991.476943]  [<ffffffff81114574>] ? lookup_fast+0x3d/0x215
[4583991.476954]  [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs]
[4583991.476966]  [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs]
[4583991.476970]  [<ffffffff81115fda>] ? vfs_create+0x62/0x8b
[4583991.476974]  [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95
[4583991.476978]  [<ffffffff81117179>] ? do_last+0x595/0xa16
[4583991.476982]  [<ffffffff811176be>] ? path_openat+0xc4/0x335
[4583991.476985]  [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e
[4583991.476989]  [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1
[4583991.476993]  [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0
[4583991.476996]  [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b

That one isn't so easy to fix, unfortunately.

Cheers,

Dave.

Signature · 14 lines
Dave Chinner
[email protected]

xfs: xfs_dir2_block_to_sf temp buffer allocation fails 

From: Dave Chinner <[email protected]>

If we are using a large directory block size, and memory becomes
fragmented, we can get memory allocation failures trying to
kmem_alloc(64k) for a temporary buffer. However, there is not need
for a directory buffer sized allocation, as the end result ends up
in the inode literal area. This is, at most, slightly less than 2k
of space, and hence we don't need an allocation larger than that
fora temporary buffer.
Quoted 8 lines
Quoted 1 line

D'oh, I missed this one too. If you stick 'patch' in the subject they'll have additional visibility.

Looks good to me.

When I reply in line like this it's more of a case of "please test this patch to see if it fixes your problem" question, not really an official posting of a patch because it's very likely I have just written the patch and have only done a 5 minute smoke test of the patch. So in this context, it's not really a "please review and commit" request.

The issue is that I haven't reposted the patch in a separate series asking for reviews and commit as I normally do after I've tested it properly and are happy with it.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs