XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Emmanuel Lacour
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in
kmem_alloc(mode:0x250)does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Dave Chinner
Quoted 9 lines · Emmanuel Lacour
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in
kmem_alloc(mode:0x250)
Quoted 8 lines
Quoted 2 lines
does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:
http://people.easter-eggs.org/~manu/xfs.log
This may be related to a friend problem here:
http://tracker.ceph.com/issues/6386
thanks for any help on solving this!
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in
kmem_alloc(mode:0x250)does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:
Quoted 8 lines
Quoted 2 lines
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.
Yup, there's your problem:
[4583991.478469] ceph-osd D ffff88047fc93f40 0 22951 1 0x00000004 [4583991.478471] ffff88046d241140 0000000000000082 ffffffff81047e75 ffff88046f949800 [4583991.478475] 0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8 ffff88046d241140 [4583991.478479] 0000000000000000 00000001444d68bd ffff88046d241140 0000000000000005 [4583991.478483] Call Trace: [4583991.478487] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.478491] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.478494] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.478498] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.478502] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.478505] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.478518] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.478535] [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs] [4583991.478550] [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs] [4583991.478566] [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs] [4583991.478581] [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs] [4583991.478594] [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs] [4583991.478606] [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs] [4583991.478610] [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f [4583991.478614] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.478618] [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257 [4583991.478622] [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187 [4583991.478625] [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce [4583991.478629] [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68 [4583991.478633] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b
It'll be stuck on this:
hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);
which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.
Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!
I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.
Quoted 3 lines
This may be related to a friend problem here:
Doesn't look related, unless the OOM killer is being triggered somehow...
Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:
[4583991.476833] ceph-osd D ffff88047fc33f40 0 11072 1 0x00000004 [4583991.476836] ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040 [4583991.476840] 0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040 [4583991.476844] 0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005 [4583991.476848] Call Trace: [4583991.476852] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.476855] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.476859] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.476862] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.476867] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.476870] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.476883] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.476899] [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs] [4583991.476904] [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30 [4583991.476909] [<ffffffff81041942>] ? current_fs_time+0x27/0x2d [4583991.476925] [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs] [4583991.476939] [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs] [4583991.476943] [<ffffffff81114574>] ? lookup_fast+0x3d/0x215 [4583991.476954] [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs] [4583991.476966] [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs] [4583991.476970] [<ffffffff81115fda>] ? vfs_create+0x62/0x8b [4583991.476974] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.476978] [<ffffffff81117179>] ? do_last+0x595/0xa16 [4583991.476982] [<ffffffff811176be>] ? path_openat+0xc4/0x335 [4583991.476985] [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e [4583991.476989] [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1 [4583991.476993] [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0 [4583991.476996] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b
That one isn't so easy to fix, unfortunately.
Cheers,
Dave.
Signature · 19 lines
Dave Chinner [email protected] xfs: xfs_dir2_block_to_sf temp buffer allocation fails From: Dave Chinner <[email protected]> If we are using a large directory block size, and memory becomes fragmented, we can get memory allocation failures trying to kmem_alloc(64k) for a temporary buffer. However, there is not need for a directory buffer sized allocation, as the end result ends up in the inode literal area. This is, at most, slightly less than 2k of space, and hence we don't need an allocation larger than that fora temporary buffer. Signed-off-by: Dave Chinner <[email protected]> --- fs/xfs/xfs_dir2_sf.c | 58 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 24 deletions(-)
fs/xfs/xfs_dir2_sf.c +34 -24
@@ -170,6 +170,7 @@ xfs_dir2_block_to_sf(char *ptr;/* current data pointer */ xfs_dir2_sf_entry_t *sfep;/* shortform entry */ xfs_dir2_sf_hdr_t *sfp;/* shortform directory header */ +xfs_dir2_sf_hdr_t *dst;/* temporary data buffer */ trace_xfs_dir2_block_to_sf (args);
@@ -177,35 +178,20 @@ xfs_dir2_block_to_sf( mp = dp->i_mount;/* -* Make a copy of the block data, so we can shrink the inode -* and add local data. +* allocate a temporary destination buffer the size of the inode +* to format the data into. Once we have formatted the data, we +* can free the block and copy the formatted data into the inode literal +* area. */ - hdr =kmem_alloc (mp->m_dirblksize,KM_SLEEP ); -memcpy (hdr, bp->b_addr, mp->m_dirblksize); - logflags =XFS_ILOG_CORE ; -if ((error =xfs_dir2_shrink_inode (args, mp->m_dirdatablk, bp))) { -ASSERT (error !=ENOSPC ); - goto out; - } + dst =kmem_alloc (mp->m_sb.sb_inodesize,KM_SLEEP ); + hdr = bp->b_addr;/* -* The buffer is now unconditionally gone, whether -* xfs_dir2_shrink_inode worked or not. -* -* Convert the inode to local format. -*/ - dp->i_df.if_flags &= ~XFS_IFEXTENTS ; - dp->i_df.if_flags |=XFS_IFINLINE ; - dp->i_d.di_format =XFS_DINODE_FMT_LOCAL ; -ASSERT (dp->i_df.if_bytes ==0 ); -xfs_idata_realloc (dp, size,XFS_DATA_FORK ); - logflags |=XFS_ILOG_DDATA ; -/* * Copy the header into the newly allocate local space. */ - sfp = (xfs_dir2_sf_hdr_t *)dp->i_df.if_u1.if_data; + sfp = (xfs_dir2_sf_hdr_t *)dst;memcpy (sfp, sfhp,xfs_dir2_sf_hdr_size (sfhp->i8count)); - dp->i_d.di_size = size; +/* * Set up to loop over the block's entries. */
@@ -258,10 +244,34 @@ xfs_dir2_block_to_sf( ptr += dp->d_ops->data_entsize (dep->namelen); }ASSERT ((char *)sfep - (char *)sfp == size); + +/* now we are done with the block, we can shrink the inode */ + logflags =XFS_ILOG_CORE ; + error =xfs_dir2_shrink_inode (args, mp->m_dirdatablk, bp); +if (error) { +ASSERT (error !=ENOSPC ); + goto out; + } + +/* +* The buffer is now unconditionally gone, whether +* xfs_dir2_shrink_inode worked or not. +* +* Convert the inode to local format and copy the data in. +*/ + dp->i_df.if_flags &= ~XFS_IFEXTENTS ; + dp->i_df.if_flags |=XFS_IFINLINE ; + dp->i_d.di_format =XFS_DINODE_FMT_LOCAL ; +ASSERT (dp->i_df.if_bytes ==0 ); +xfs_idata_realloc (dp, size,XFS_DATA_FORK ); + + logflags |=XFS_ILOG_DDATA ; +memcpy (dp->i_df.if_u1.if_data, dst, size); + dp->i_d.di_size = size;xfs_dir2_sf_check (args); out:xfs_trans_log_inode (args->trans, dp, logflags); -kmem_free (hdr); +kmem_free (dst);return error; }
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Dave Chinner
On Tue, Dec 03, 2013 at 10:53:58AM +0100, Emmanuel Lacour wrote:
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size. Yup, there's your problem:
I can confirm I also saw this from times to times, in the very same context (ceph OSD, xfs volume with 64k directory blocks)
I never took the time to fully report the problem because it was very sporadic, and I suspected a specific option in my hand-made kernels, in which the problem seems to occurs more often. With the 'standard' debian 3.10.2 (from testing) I never saw the problem (56 days uptime)
cf http://tracker.ceph.com/issues/6301
the report isn't complete here ... sometimes I just have the deadlock, but without oops. can try to dig deeper inmy old logs to be sure it's really the same
Cheers,
Signature · 7 lines
Yann Dupont - Service IRTS, DSI Université de Nantes Tel : 02.53.48.49.20 - Mail/Jabber : [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · Dave Chinner
thanks very much for your quick and detailled answer!
Quoted 4 lines
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.
yes, the 64k was taken from a too quickly read of advice, I don't think it's of any help on a ceph cluster but I'm not an FS guru. Is there a way to lower it at runtime?
Quoted 2 lines
Yup, there's your problem:
[...]
Quoted 4 lines
Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!
unfortunatly I cannot test this patch because:
- it's a production cluster and it's currently hard for me to reboot nodes (not enough nodes ;))
- just after hiting this problem I saw a kernel 3.11 available on Debian backports and decided to upgrade the whole cluster.
since this upgrade, there is no problems anymore ... I cross my fingers ;)
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Ben Myers
Reviewed-byQuoted 121 lines · Dave Chinner
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in
kmem_alloc(mode:0x250)does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:
Quoted 8 lines
Quoted 2 lines
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.
Yup, there's your problem:
[4583991.478469] ceph-osd D ffff88047fc93f40 0 22951 1 0x00000004 [4583991.478471] ffff88046d241140 0000000000000082 ffffffff81047e75 ffff88046f949800 [4583991.478475] 0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8 ffff88046d241140 [4583991.478479] 0000000000000000 00000001444d68bd ffff88046d241140 0000000000000005 [4583991.478483] Call Trace: [4583991.478487] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.478491] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.478494] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.478498] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.478502] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.478505] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.478518] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.478535] [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs] [4583991.478550] [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs] [4583991.478566] [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs] [4583991.478581] [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs] [4583991.478594] [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs] [4583991.478606] [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs] [4583991.478610] [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f [4583991.478614] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.478618] [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257 [4583991.478622] [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187 [4583991.478625] [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce [4583991.478629] [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68 [4583991.478633] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1bIt'll be stuck on this:
hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.
Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!
I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.
Quoted 3 lines
This may be related to a friend problem here:
Doesn't look related, unless the OOM killer is being triggered somehow...
Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:
[4583991.476833] ceph-osd D ffff88047fc33f40 0 11072 1 0x00000004 [4583991.476836] ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040 [4583991.476840] 0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040 [4583991.476844] 0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005 [4583991.476848] Call Trace: [4583991.476852] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.476855] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.476859] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.476862] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.476867] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.476870] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.476883] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.476899] [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs] [4583991.476904] [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30 [4583991.476909] [<ffffffff81041942>] ? current_fs_time+0x27/0x2d [4583991.476925] [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs] [4583991.476939] [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs] [4583991.476943] [<ffffffff81114574>] ? lookup_fast+0x3d/0x215 [4583991.476954] [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs] [4583991.476966] [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs] [4583991.476970] [<ffffffff81115fda>] ? vfs_create+0x62/0x8b [4583991.476974] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.476978] [<ffffffff81117179>] ? do_last+0x595/0xa16 [4583991.476982] [<ffffffff811176be>] ? path_openat+0xc4/0x335 [4583991.476985] [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e [4583991.476989] [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1 [4583991.476993] [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0 [4583991.476996] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1b
Quoted 22 lines
That one isn't so easy to fix, unfortunately.
Cheers,
Dave.
Signature · 16 lines
Dave Chinner [email protected] xfs: xfs_dir2_block_to_sf temp buffer allocation fails From: Dave Chinner <[email protected]> If we are using a large directory block size, and memory becomes fragmented, we can get memory allocation failures trying to kmem_alloc(64k) for a temporary buffer. However, there is not need for a directory buffer sized allocation, as the end result ends up in the inode literal area. This is, at most, slightly less than 2k of space, and hence we don't need an allocation larger than that fora temporary buffer. Signed-off-by: Dave Chinner <[email protected]>
D'oh, I missed this one too. If you stick 'patch' in the subject they'll have additional visibility.
Looks good to me.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 143 lines · Ben Myers
Quoted 142 lines · Dave Chinner
Quoted 17 lines · Emmanuel Lacour
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Emmanuel Lacour
Dear XFS users,
I run a Ceph cluster using XFS on Debian wheezy servers and Linux 3.10 (debian backports). I see the following line in our logs:
XFS: possible memory allocation deadlock in
kmem_alloc(mode:0x250)does this reveal a problem in my setup or may I ignore it? If it's a problem, can someone give me any hint on solving this?
It might be, but you need to provide more information for us to be able to make any intelligent comment on the message. Start here:
Quoted 8 lines
Quoted 2 lines
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
The problem continue and crashed my ceph cluster again, so here is all the informations said in the FAQ:
OK, 32GB RAM, no obvious shortage, no dirty or writeback data. 2TB SATA drives, 32AGs, only unusual setting is 64k directory block size.
Yup, there's your problem:
[4583991.478469] ceph-osd D ffff88047fc93f40 0 22951 1 0x00000004 [4583991.478471] ffff88046d241140 0000000000000082 ffffffff81047e75 ffff88046f949800 [4583991.478475] 0000000000013f40 ffff88039eb0bfd8 ffff88039eb0bfd8 ffff88046d241140 [4583991.478479] 0000000000000000 00000001444d68bd ffff88046d241140 0000000000000005 [4583991.478483] Call Trace: [4583991.478487] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.478491] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.478494] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.478498] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.478502] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.478505] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.478518] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.478535] [<ffffffffa0592c4a>] ? xfs_dir2_block_to_sf+0x5b/0x1fb [xfs] [4583991.478550] [<ffffffffa0592be0>] ? xfs_dir2_block_sfsize+0x15b/0x16a [xfs] [4583991.478566] [<ffffffffa058bf9a>] ? xfs_dir2_block_removename+0x1c7/0x208 [xfs] [4583991.478581] [<ffffffffa058ab4a>] ? xfs_dir_removename+0xda/0x114 [xfs] [4583991.478594] [<ffffffffa056a55c>] ? xfs_rename+0x428/0x554 [xfs] [4583991.478606] [<ffffffffa0567321>] ? xfs_vn_rename+0x5e/0x65 [xfs] [4583991.478610] [<ffffffff8111677b>] ? vfs_rename+0x224/0x35f [4583991.478614] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.478618] [<ffffffff81116a7e>] ? SYSC_renameat+0x1c8/0x257 [4583991.478622] [<ffffffff810fb0fd>] ? __cache_free.isra.45+0x178/0x187 [4583991.478625] [<ffffffff81117eb1>] ? SyS_mkdirat+0x2e/0xce [4583991.478629] [<ffffffff8100d56a>] ? do_notify_resume+0x53/0x68 [4583991.478633] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1bIt'll be stuck on this:
hdr = kmem_alloc(mp->m_dirblksize, KM_SLEEP);which is trying to allocate a contiguous 64k buffer to copy the direct contents into before freeing the block and then formatting them into the inode. The failure will be caused by memory fragmentation, and the only way around it is to avoid the contiguous allocation of that size.
Which, I think, is pretty easy to do. Yup, barely smoke tested patch below that demonstrates the fix. Beware - patch may eat babies and ask for more. Use it at your own risk!
I'll post it for review once it's had some testing and I know it doesn't corrupt directories all over the place.
Quoted 3 lines
This may be related to a friend problem here:
Doesn't look related, unless the OOM killer is being triggered somehow...
Hmmmm - there's also a good chance the the transaction commit code has this same problem contiguous allocation problem given that it has to allocate enough space to log an entire directory buffer. Good guess - there's another thread stuck on exactly that:
[4583991.476833] ceph-osd D ffff88047fc33f40 0 11072 1 0x00000004 [4583991.476836] ffff88038b32a040 0000000000000082 ffffffff81047e75 ffff88046f946040 [4583991.476840] 0000000000013f40 ffff88048ea11fd8 ffff88048ea11fd8 ffff88038b32a040 [4583991.476844] 0000000000000000 00000001444d68be ffff88038b32a040 0000000000000005 [4583991.476848] Call Trace: [4583991.476852] [<ffffffff81047e75>] ? internal_add_timer+0xd/0x28 [4583991.476855] [<ffffffff8138e34a>] ? schedule_timeout+0xeb/0x123 [4583991.476859] [<ffffffff81047e63>] ? ftrace_raw_event_timer_class+0x9d/0x9d [4583991.476862] [<ffffffff8138edb6>] ? io_schedule_timeout+0x60/0x86 [4583991.476867] [<ffffffff810d85ad>] ? congestion_wait+0x70/0xdb [4583991.476870] [<ffffffff8105858f>] ? abort_exclusive_wait+0x79/0x79 [4583991.476883] [<ffffffffa056e3f9>] ? kmem_alloc+0x65/0x6f [xfs] [4583991.476899] [<ffffffffa05a77a6>] ? xfs_log_commit_cil+0xe8/0x3d1 [xfs] [4583991.476904] [<ffffffff810748ab>] ? current_kernel_time+0x9/0x30 [4583991.476909] [<ffffffff81041942>] ? current_fs_time+0x27/0x2d [4583991.476925] [<ffffffffa05a3b7b>] ? xfs_trans_commit+0x62/0x1cf [xfs] [4583991.476939] [<ffffffffa056d3ad>] ? xfs_create+0x41e/0x54f [xfs] [4583991.476943] [<ffffffff81114574>] ? lookup_fast+0x3d/0x215 [4583991.476954] [<ffffffffa056cf29>] ? xfs_lookup+0x88/0xee [xfs] [4583991.476966] [<ffffffffa0567428>] ? xfs_vn_mknod+0xb7/0x162 [xfs] [4583991.476970] [<ffffffff81115fda>] ? vfs_create+0x62/0x8b [4583991.476974] [<ffffffff81113d0b>] ? lookup_dcache+0x22/0x95 [4583991.476978] [<ffffffff81117179>] ? do_last+0x595/0xa16 [4583991.476982] [<ffffffff811176be>] ? path_openat+0xc4/0x335 [4583991.476985] [<ffffffff81117bda>] ? do_filp_open+0x2a/0x6e [4583991.476989] [<ffffffff81120b62>] ? __alloc_fd+0xd0/0xe1 [4583991.476993] [<ffffffff8110b684>] ? do_sys_open+0x5c/0xe0 [4583991.476996] [<ffffffff81395429>] ? system_call_fastpath+0x16/0x1bThat one isn't so easy to fix, unfortunately.
Cheers,
Dave.
Signature · 14 lines
Dave Chinner [email protected] xfs: xfs_dir2_block_to_sf temp buffer allocation fails From: Dave Chinner <[email protected]> If we are using a large directory block size, and memory becomes fragmented, we can get memory allocation failures trying to kmem_alloc(64k) for a temporary buffer. However, there is not need for a directory buffer sized allocation, as the end result ends up in the inode literal area. This is, at most, slightly less than 2k of space, and hence we don't need an allocation larger than that fora temporary buffer.
Quoted 8 lines
Quoted 1 line
Signed-off-by Dave Chinner <[email protected]>D'oh, I missed this one too. If you stick 'patch' in the subject they'll have additional visibility.
Looks good to me.
Reviewed-by Ben Myers <[email protected]>
When I reply in line like this it's more of a case of "please test this patch to see if it fixes your problem" question, not really an official posting of a patch because it's very likely I have just written the patch and have only done a 5 minute smoke test of the patch. So in this context, it's not really a "please review and commit" request.
The issue is that I haven't reposted the patch in a separate series asking for reviews and commit as I normally do after I've tested it properly and are happy with it.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs