Failing XFS memory allocation

Hello,

So I have an XFS filesystem which houses 2 2.3T sparse files, which are loop-mounted. Recently I migrated a server to a 4.4.6 kernel and this morning I observed the following in my dmesg:

XFS: loop0(15174) possible memory allocation deadlock size 107168 in kmem_alloc (mode:0x2400240)

the mode is essentially (GFP_KERNEL | GFP_NOWARN) &= ~__GFP_FS. Here is the site of the loop file in case it matters:

du -h --apparent-size /storage/loop/file1
2.3T /storage/loop/file1

du -h /storage/loop/file1
878G /storage/loop/file1

And this string is repeated multiple times. Looking at the output of "echo w > /proc/sysrq-trigger" I see the following suspicious entry:

loop0 D ffff881fe081f038 0 15174 2 0x00000000
ffff881fe081f038 ffff883ff29fa700 ffff881fecb70d00 ffff88407fffae00 0000000000000000 0000000502404240 ffffffff81e30d60 0000000000000000 0000000000000000 ffff881f00000003 0000000000000282 ffff883f00000000

Call Trace:
 [<ffffffff8163ac01>] ? _raw_spin_lock_irqsave+0x21/0x60
 [<ffffffff81636fd7>] schedule+0x47/0x90
 [<ffffffff81639f03>] schedule_timeout+0x113/0x1e0
 [<ffffffff810ac580>] ? lock_timer_base+0x80/0x80
 [<ffffffff816363d4>] io_schedule_timeout+0xa4/0x110
 [<ffffffff8114aadf>] congestion_wait+0x7f/0x130
 [<ffffffff810939e0>] ? woken_wake_function+0x20/0x20
 [<ffffffffa0283bac>] kmem_alloc+0x8c/0x120 [xfs]
 [<ffffffff81181751>] ? __kmalloc+0x121/0x250
 [<ffffffffa0283c73>] kmem_realloc+0x33/0x80 [xfs]
 [<ffffffffa02546cd>] xfs_iext_realloc_indirect+0x3d/0x60 [xfs]
 [<ffffffffa02548cf>] xfs_iext_irec_new+0x3f/0xf0 [xfs]
 [<ffffffffa0254c0d>] xfs_iext_add_indirect_multi+0x14d/0x210 [xfs]
 [<ffffffffa02554b5>] xfs_iext_add+0xc5/0x230 [xfs]

[<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20

 [<ffffffffa0256269>] xfs_iext_insert+0x59/0x110 [xfs]
 [<ffffffffa0230928>] ? xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffffa0230928>] xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff8112b725>] ? mempool_alloc+0x65/0x180
 [<ffffffffa02543d8>] ? xfs_iext_get_ext+0x38/0x70 [xfs]
 [<ffffffffa0254e8d>] ? xfs_iext_bno_to_ext+0xed/0x150 [xfs]
 [<ffffffffa02311b5>] xfs_bmapi_reserve_delalloc+0x225/0x250 [xfs]
 [<ffffffffa023131e>] xfs_bmapi_delay+0x13e/0x290 [xfs]
 [<ffffffffa02730ad>] xfs_iomap_write_delay+0x17d/0x300 [xfs]
 [<ffffffffa022e434>] ? xfs_bmapi_read+0x114/0x330 [xfs]
 [<ffffffffa025ddc5>] __xfs_get_blocks+0x585/0xa90 [xfs]
 [<ffffffff81324b53>] ? __percpu_counter_add+0x63/0x80
 [<ffffffff811374cd>] ? account_page_dirtied+0xed/0x1b0
 [<ffffffff811cfc59>] ? alloc_buffer_head+0x49/0x60
 [<ffffffff811d07c0>] ? alloc_page_buffers+0x60/0xb0
 [<ffffffff811d13e5>] ? create_empty_buffers+0x45/0xc0
 [<ffffffffa025e324>] xfs_get_blocks+0x14/0x20 [xfs]
 [<ffffffff811d34e2>] __block_write_begin+0x1c2/0x580
 [<ffffffffa025e310>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
 [<ffffffffa025bbb1>] xfs_vm_write_begin+0x61/0xf0 [xfs]
 [<ffffffff81127e50>] generic_perform_write+0xd0/0x1f0
 [<ffffffffa026a341>] xfs_file_buffered_aio_write+0xe1/0x240 [xfs]
 [<ffffffff812e16d2>] ? bt_clear_tag+0xb2/0xd0
 [<ffffffffa026ab87>] xfs_file_write_iter+0x167/0x170 [xfs]
 [<ffffffff81199d76>] vfs_iter_write+0x76/0xa0
 [<ffffffffa03fb735>] lo_write_bvec+0x65/0x100 [loop]
 [<ffffffffa03fd589>] loop_queue_work+0x689/0x924 [loop]
 [<ffffffff8163ba52>] ? retint_kernel+0x10/0x10
 [<ffffffff81074d71>] kthread_worker_fn+0x61/0x1c0
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff810744d7>] kthread+0xd7/0xf0
 [<ffffffff8107d22e>] ? schedule_tail+0x1e/0xd0
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80
 [<ffffffff8163b2af>] ret_from_fork+0x3f/0x70
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80

So this seems that there are writes to the loop device being queued and while being served XFS has to do some internal memory allocation to fit the new data, however due to some *uknown* reason it fails and starts looping in kmem_alloc. I didn't see any OOM reports so presumably the server was not out of memory, but unfortunately I didn't check the memory fragmentation, though I collected a crash dump in case you need further info.

The one thing which bugs me is that XFS tried to allocate 107 contiguous kb which is page-order-26 isn't this waaaaay too big and almost never satisfiable, despite direct/bg reclaim to be enabled? For now I've reverted to using 3.12.52 kernel, where this issue hasn't been observed (yet) any ideas would be much appreciated.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 9 lines · Nikolay Borisov

Hello,

So I have an XFS filesystem which houses 2 2.3T sparse files, which are loop-mounted. Recently I migrated a server to a 4.4.6 kernel and this morning I observed the following in my dmesg:

XFS: loop0(15174) possible memory allocation deadlock size 107168 in kmem_alloc (mode:0x2400240)

Is there a stack trace associated with this message?

Quoted 16 lines

the mode is essentially (GFP_KERNEL | GFP_NOWARN) &= ~__GFP_FS. Here is the site of the loop file in case it matters:

du -h --apparent-size /storage/loop/file1
2.3T /storage/loop/file1

du -h /storage/loop/file1
878G /storage/loop/file1

And this string is repeated multiple times. Looking at the output of "echo w > /proc/sysrq-trigger" I see the following suspicious entry:

loop0 D ffff881fe081f038 0 15174 2 0x00000000
ffff881fe081f038 ffff883ff29fa700 ffff881fecb70d00 ffff88407fffae00 0000000000000000 0000000502404240 ffffffff81e30d60 0000000000000000 0000000000000000 ffff881f00000003 0000000000000282 ffff883f00000000

Quoted 15 lines

Call Trace:
 [<ffffffff8163ac01>] ? _raw_spin_lock_irqsave+0x21/0x60
 [<ffffffff81636fd7>] schedule+0x47/0x90
 [<ffffffff81639f03>] schedule_timeout+0x113/0x1e0
 [<ffffffff810ac580>] ? lock_timer_base+0x80/0x80
 [<ffffffff816363d4>] io_schedule_timeout+0xa4/0x110
 [<ffffffff8114aadf>] congestion_wait+0x7f/0x130
 [<ffffffff810939e0>] ? woken_wake_function+0x20/0x20
 [<ffffffffa0283bac>] kmem_alloc+0x8c/0x120 [xfs]
 [<ffffffff81181751>] ? __kmalloc+0x121/0x250
 [<ffffffffa0283c73>] kmem_realloc+0x33/0x80 [xfs]
 [<ffffffffa02546cd>] xfs_iext_realloc_indirect+0x3d/0x60 [xfs]
 [<ffffffffa02548cf>] xfs_iext_irec_new+0x3f/0xf0 [xfs]
 [<ffffffffa0254c0d>] xfs_iext_add_indirect_multi+0x14d/0x210 [xfs]
 [<ffffffffa02554b5>] xfs_iext_add+0xc5/0x230 [xfs]

It looks like it's working to add a new extent to the in-core extent list. If this is the stack associated with the warning message (combined with the large alloc size), I wonder if there's a fragmentation issue on the file leading to an excessive number of extents.

What does 'xfs_bmap -v /storage/loop/file1' show?

Brian

Quoted 57 lines

[<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20

 [<ffffffffa0256269>] xfs_iext_insert+0x59/0x110 [xfs]
 [<ffffffffa0230928>] ? xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffffa0230928>] xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff8112b725>] ? mempool_alloc+0x65/0x180
 [<ffffffffa02543d8>] ? xfs_iext_get_ext+0x38/0x70 [xfs]
 [<ffffffffa0254e8d>] ? xfs_iext_bno_to_ext+0xed/0x150 [xfs]
 [<ffffffffa02311b5>] xfs_bmapi_reserve_delalloc+0x225/0x250 [xfs]
 [<ffffffffa023131e>] xfs_bmapi_delay+0x13e/0x290 [xfs]
 [<ffffffffa02730ad>] xfs_iomap_write_delay+0x17d/0x300 [xfs]
 [<ffffffffa022e434>] ? xfs_bmapi_read+0x114/0x330 [xfs]
 [<ffffffffa025ddc5>] __xfs_get_blocks+0x585/0xa90 [xfs]
 [<ffffffff81324b53>] ? __percpu_counter_add+0x63/0x80
 [<ffffffff811374cd>] ? account_page_dirtied+0xed/0x1b0
 [<ffffffff811cfc59>] ? alloc_buffer_head+0x49/0x60
 [<ffffffff811d07c0>] ? alloc_page_buffers+0x60/0xb0
 [<ffffffff811d13e5>] ? create_empty_buffers+0x45/0xc0
 [<ffffffffa025e324>] xfs_get_blocks+0x14/0x20 [xfs]
 [<ffffffff811d34e2>] __block_write_begin+0x1c2/0x580
 [<ffffffffa025e310>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
 [<ffffffffa025bbb1>] xfs_vm_write_begin+0x61/0xf0 [xfs]
 [<ffffffff81127e50>] generic_perform_write+0xd0/0x1f0
 [<ffffffffa026a341>] xfs_file_buffered_aio_write+0xe1/0x240 [xfs]
 [<ffffffff812e16d2>] ? bt_clear_tag+0xb2/0xd0
 [<ffffffffa026ab87>] xfs_file_write_iter+0x167/0x170 [xfs]
 [<ffffffff81199d76>] vfs_iter_write+0x76/0xa0
 [<ffffffffa03fb735>] lo_write_bvec+0x65/0x100 [loop]
 [<ffffffffa03fd589>] loop_queue_work+0x689/0x924 [loop]
 [<ffffffff8163ba52>] ? retint_kernel+0x10/0x10
 [<ffffffff81074d71>] kthread_worker_fn+0x61/0x1c0
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff810744d7>] kthread+0xd7/0xf0
 [<ffffffff8107d22e>] ? schedule_tail+0x1e/0xd0
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80
 [<ffffffff8163b2af>] ret_from_fork+0x3f/0x70
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80

So this seems that there are writes to the loop device being queued and while being served XFS has to do some internal memory allocation to fit the new data, however due to some *uknown* reason it fails and starts looping in kmem_alloc. I didn't see any OOM reports so presumably the server was not out of memory, but unfortunately I didn't check the memory fragmentation, though I collected a crash dump in case you need further info.

The one thing which bugs me is that XFS tried to allocate 107 contiguous kb which is page-order-26 isn't this waaaaay too big and almost never satisfiable, despite direct/bg reclaim to be enabled? For now I've reverted to using 3.12.52 kernel, where this issue hasn't been observed (yet) any ideas would be much appreciated.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 1 line · Brian Foster

On Wed, Mar 23, 2016 at 12:15:42PM +0200, Nikolay Borisov wrote:

Quoted 4 lines

It looks like it's working to add a new extent to the in-core extent list. If this is the stack associated with the warning message (combined with the large alloc size), I wonder if there's a fragmentation issue on the file leading to an excessive number of extents.

Yes this is the stack trace associated.

Quoted 2 lines

What does 'xfs_bmap -v /storage/loop/file1' show?

It spews a lot of stuff but here is a summary, more detailed info can be provided if you need it:

xfs_bmap -v /storage/loop/file1 | wc -l
900908
xfs_bmap -v /storage/loop/file1 | grep -c hole
94568

Also, what would constitute an "excessive number of extents"?

Quoted 60 lines

Brian

Quoted 57 lines

[<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20

 [<ffffffffa0256269>] xfs_iext_insert+0x59/0x110 [xfs]
 [<ffffffffa0230928>] ? xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffffa0230928>] xfs_bmap_add_extent_hole_delay+0xd8/0x740 [xfs]
 [<ffffffff8112b5c5>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff8112b725>] ? mempool_alloc+0x65/0x180
 [<ffffffffa02543d8>] ? xfs_iext_get_ext+0x38/0x70 [xfs]
 [<ffffffffa0254e8d>] ? xfs_iext_bno_to_ext+0xed/0x150 [xfs]
 [<ffffffffa02311b5>] xfs_bmapi_reserve_delalloc+0x225/0x250 [xfs]
 [<ffffffffa023131e>] xfs_bmapi_delay+0x13e/0x290 [xfs]
 [<ffffffffa02730ad>] xfs_iomap_write_delay+0x17d/0x300 [xfs]
 [<ffffffffa022e434>] ? xfs_bmapi_read+0x114/0x330 [xfs]
 [<ffffffffa025ddc5>] __xfs_get_blocks+0x585/0xa90 [xfs]
 [<ffffffff81324b53>] ? __percpu_counter_add+0x63/0x80
 [<ffffffff811374cd>] ? account_page_dirtied+0xed/0x1b0
 [<ffffffff811cfc59>] ? alloc_buffer_head+0x49/0x60
 [<ffffffff811d07c0>] ? alloc_page_buffers+0x60/0xb0
 [<ffffffff811d13e5>] ? create_empty_buffers+0x45/0xc0
 [<ffffffffa025e324>] xfs_get_blocks+0x14/0x20 [xfs]
 [<ffffffff811d34e2>] __block_write_begin+0x1c2/0x580
 [<ffffffffa025e310>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
 [<ffffffffa025bbb1>] xfs_vm_write_begin+0x61/0xf0 [xfs]
 [<ffffffff81127e50>] generic_perform_write+0xd0/0x1f0
 [<ffffffffa026a341>] xfs_file_buffered_aio_write+0xe1/0x240 [xfs]
 [<ffffffff812e16d2>] ? bt_clear_tag+0xb2/0xd0
 [<ffffffffa026ab87>] xfs_file_write_iter+0x167/0x170 [xfs]
 [<ffffffff81199d76>] vfs_iter_write+0x76/0xa0
 [<ffffffffa03fb735>] lo_write_bvec+0x65/0x100 [loop]
 [<ffffffffa03fd589>] loop_queue_work+0x689/0x924 [loop]
 [<ffffffff8163ba52>] ? retint_kernel+0x10/0x10
 [<ffffffff81074d71>] kthread_worker_fn+0x61/0x1c0
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff81074d10>] ? flush_kthread_work+0x120/0x120
 [<ffffffff810744d7>] kthread+0xd7/0xf0
 [<ffffffff8107d22e>] ? schedule_tail+0x1e/0xd0
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80
 [<ffffffff8163b2af>] ret_from_fork+0x3f/0x70
 [<ffffffff81074400>] ? kthread_freezable_should_stop+0x80/0x80

So this seems that there are writes to the loop device being queued and while being served XFS has to do some internal memory allocation to fit the new data, however due to some *uknown* reason it fails and starts looping in kmem_alloc. I didn't see any OOM reports so presumably the server was not out of memory, but unfortunately I didn't check the memory fragmentation, though I collected a crash dump in case you need further info.

The one thing which bugs me is that XFS tried to allocate 107 contiguous kb which is page-order-26 isn't this waaaaay too big and almost never satisfiable, despite direct/bg reclaim to be enabled? For now I've reverted to using 3.12.52 kernel, where this issue hasn't been observed (yet) any ideas would be much appreciated.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 4 lines · Nikolay Borisov

Quoted 1 line · Brian Foster

On Wed, Mar 23, 2016 at 12:15:42PM +0200, Nikolay Borisov wrote:

...

Quoted 13 lines

Quoted 4 lines

It looks like it's working to add a new extent to the in-core extent list. If this is the stack associated with the warning message (combined with the large alloc size), I wonder if there's a fragmentation issue on the file leading to an excessive number of extents.

Yes this is the stack trace associated.

Quoted 2 lines

What does 'xfs_bmap -v /storage/loop/file1' show?

It spews a lot of stuff but here is a summary, more detailed info can be provided if you need it:

Quoted 7 lines

xfs_bmap -v /storage/loop/file1 | wc -l
900908
xfs_bmap -v /storage/loop/file1 | grep -c hole
94568

Also, what would constitute an "excessive number of extents"?

I'm not sure where one would draw the line tbh, it's just a matter of having too many extents to the point that it causes problems in terms of performance (i.e., reading/modifying the extent list) or such as the allocation problem you're running into. As it is, XFS maintains the full extent list for an active inode in memory, so that's 800k+ extents that it's looking for memory for.

It looks like that is your problem here. 800k or so extents over 878G looks to be about 1MB per extent. Are you using extent size hints? One option that might prevent this is to use a larger extent size hint value. Another might be to preallocate the entire file up front with fallocate. You'd probably have to experiment with what option or value works best for your workload.

Quoted 6 lines

I'm not sure where one would draw the line tbh, it's just a matter of having too many extents to the point that it causes problems in terms of performance (i.e., reading/modifying the extent list) or such as the allocation problem you're running into. As it is, XFS maintains the full extent list for an active inode in memory, so that's 800k+ extents that it's looking for memory for.

I saw in the comments that this problem has already been identified and a possible solution would be to add another level of indirection. Also, can you confirm that my understanding of the operation of the indirection array is correct in that each entry in the indirection array xfs_ext_irec is responsible for 256 extents. (the er_extbuf is PAGE_SIZE/4kb and an extent is 16 bytes which results in 256 extents)

Quoted 7 lines

It looks like that is your problem here. 800k or so extents over 878G looks to be about 1MB per extent. Are you using extent size hints? One option that might prevent this is to use a larger extent size hint value. Another might be to preallocate the entire file up front with fallocate. You'd probably have to experiment with what option or value works best for your workload.

By preallocating with fallocate you mean using fallocate with FALLOC_FL_ZERO_RANGE and not FALLOC_FL_PUNCH_HOLE, right? Because as it stands now the file does have holes, which presumably are being filled and in order to be filled an extent has to be allocated which caused the issue? Am I right in this reasoning?

Currently I'm not using extents size hint but will look into that, also if the extent size hint is say 4mb, wouldn't that cause a fairly serious loss of space, provided that the writes are smaller than 4mb. Would XFS try to perform some sort of extent coalescing or something else? I'm not an FS developer but my understanding is that with a 4mb extent size, whenever a new write occurs even if it's 256kb a new 4mb extent would be allocated, no?

And a final question - when i printed the contents of the inode with xfs_db I get core.nextents = 972564 whereas invoking the xfs_bmap | wc -l on the file always gives varying numbers?

Thanks a lot for taking the time to reply.

Quoted 2 lines · Nikolay Borisov

Quoted 1 line · Brian Foster

On Wed, Mar 23, 2016 at 12:15:42PM +0200, Nikolay Borisov wrote:

...

Quoted 13 lines

Quoted 4 lines

It looks like it's working to add a new extent to the in-core extent list. If this is the stack associated with the warning message (combined with the large alloc size), I wonder if there's a fragmentation issue on the file leading to an excessive number of extents.

Yes this is the stack trace associated.

Quoted 2 lines

What does 'xfs_bmap -v /storage/loop/file1' show?

It spews a lot of stuff but here is a summary, more detailed info can be provided if you need it:

Quoted 7 lines

xfs_bmap -v /storage/loop/file1 | wc -l
900908
xfs_bmap -v /storage/loop/file1 | grep -c hole
94568

Also, what would constitute an "excessive number of extents"?

I'm not sure where one would draw the line tbh, it's just a matter of having too many extents to the point that it causes problems in terms of performance (i.e., reading/modifying the extent list) or such as the allocation problem you're running into. As it is, XFS maintains the full extent list for an active inode in memory, so that's 800k+ extents that it's looking for memory for.

It looks like that is your problem here. 800k or so extents over 878G looks to be about 1MB per extent.

Quoted 32 lines · Brian Foster

Quoted 2 lines · Nikolay Borisov

Quoted 1 line · Brian Foster

On Wed, Mar 23, 2016 at 12:15:42PM +0200, Nikolay Borisov wrote:

...

Quoted 13 lines

Quoted 4 lines

It looks like it's working to add a new extent to the in-core extent list. If this is the stack associated with the warning message (combined with the large alloc size), I wonder if there's a fragmentation issue on the file leading to an excessive number of extents.

Yes this is the stack trace associated.

Quoted 2 lines

What does 'xfs_bmap -v /storage/loop/file1' show?

It spews a lot of stuff but here is a summary, more detailed info can be provided if you need it:

Quoted 7 lines

xfs_bmap -v /storage/loop/file1 | wc -l
900908
xfs_bmap -v /storage/loop/file1 | grep -c hole
94568

Also, what would constitute an "excessive number of extents"?

I'm not sure where one would draw the line tbh, it's just a matter of having too many extents to the point that it causes problems in terms of performance (i.e., reading/modifying the extent list) or such as the allocation problem you're running into. As it is, XFS maintains the full extent list for an active inode in memory, so that's 800k+ extents that it's looking for memory for.

Quoted 2 lines

It looks like that is your problem here. 800k or so extents over 878G looks to be about 1MB per extent.

Which I wouldn't call excessive. I use a 1MB extent size hint on all my VM images as this allows the underlying device to do IOs large enough to maintain clear to full bandwidth when reading and writing regions of the underlying image file that are non-contiguous w.r.t. sequential IO from the guest.

Mind you, it's not until I use ext4 or btrfs in the guests that I actually see significant increases in extent size. Rule of thumb in my testing is that if XFs creates 100k extents in the image file, ext4 will create 500k, and btrfs will create somewhere between 1m and 5m extents....

i.e. XFS as a guest filesystem gives results in much lower image file fragmentation that the other options....

As it is, yes, the memory allocation problem is with the in-core extent tree, and we've known about it for some time. The issue is that as memory gets fragmented, the top level indirection array grows too large to be allocated as a contiguous chunk. When this happens really depends on memory load, uptime and the way the extent tree is being modified.

I'm working on prototype patches to convert it to an in-memory btree but they are far from ready at this point. This isn't straight forward because all the extent management code assumes extents are kept in a linear array and can be directly indexed by array offset rather than file offset. I also want to make sure we can demand page the extent list if necessary, and that also complicates things like locking, as we currently assume the extent list is either completely in memory or not in memory at all.

Fundamentally, I don't want to repeat the mistakes ext4 and btrfs have made with their fine-grained in memory extent trees that are based on rb-trees (e.g. global locks, shrinkers that don't scale or consume way too much CPU, excessive memory consumption, etc) and so solving all aspects of the problem in one go is somewhat complex. And, of course, there's so much other stuff that needs to be done at the same time, I cannot find much time to work on it at the moment...

Cheers,

Dave.

Quoted 6 lines · Dave Chinner

As it is, yes, the memory allocation problem is with the in-core extent tree, and we've known about it for some time. The issue is that as memory gets fragmented, the top level indirection array grows too large to be allocated as a contiguous chunk. When this happens really depends on memory load, uptime and the way the extent tree is being modified.

And what about the following completely crazy idea of switching order > 3 allocations to using vmalloc? I know this would incur heavy performance hit, but other than that would it cause correctness issues? Of course I'm not saying this should be implemented in upstream rather whether it's worth it having a go for experimenting with this idea.

I'm working on prototype patches to convert it to an in-memory btree but they are far from ready at this point. This isn't straight forward because all the extent management code assumes extents are kept in a linear array and can be directly indexed by array offset rather than file offset. I also want to make sure we can demand page the extent list if necessary, and that also complicates things like locking, as we currently assume the extent list is either completely in memory or not in memory at all.

Quoted 8 lines · Dave Chinner

I'm working on prototype patches to convert it to an in-memory btree but they are far from ready at this point. This isn't straight forward because all the extent management code assumes extents are kept in a linear array and can be directly indexed by array offset rather than file offset. I also want to make sure we can demand page the extent list if necessary, and that also complicates things like locking, as we currently assume the extent list is either completely in memory or not in memory at all.

FYI, I did patches to get rid almost all direct extent array access a while ago, but I never bothered to post it as it seemed to much churn. Have you started that work yet or would it be useful to dust those up again?

From 4cfef0d21729704c79dc26621a254e507ea372a7 Mon Sep 17 00:00:00 2001

Hi Nikolay,

can you give the patch below a spin? While it doesn't solve the root cause it makes many typical uses of kmem_realloc behave less badly, so it should help with at least some of the less dramatic cases of very fragmented files: