XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
We've started experimenting with larger directory block sizes to avoid directory fragmentation. Everything seems to work fine, except that the log is spammed with these lovely debug messages:
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
Quoted 1 line
From looking at the code, it looks like that each of those messages (there
are thousands) equates to 100 trips through the loop. My guess is that the larger blocks require multi-page allocations which are harder to satisfy. This is with 3.10 kernel.
The hardware is something like (I can find out the exact config is you want):
32 cores 128 GB RAM LSI 9271-8i RAID (one big RAID-60 with 36 disks, partitioned)
As I hinted at earlier, we end up with pretty big directories. We can semi-reliably trigger this when we run rsync on the data between two (identical) hosts over 10GbitE.
# xfs_info /dev/sda9
meta-data=/dev/sda9 isize=256 agcount=6, agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1454213211, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=65536 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0/proc/slabinfo: https://www.copy.com/s/1x1yZFjYO2EI/slab.txt
sysrq m output: https://www.copy.com/s/mYfMYfJJl2EB/sysrq-m.txt
While I realize that the message isn't bad, it does mean that the system is having hard time allocating memory. This could potentially lead to bad performance, or even an actual deadlock. Do you have any suggestions?
Thanks,
Jeff.
Signature · 9 lines
The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Josef 'Jeff' Sipek
We've started experimenting with larger directory block sizes to avoid directory fragmentation. Everything seems to work fine, except that the log is spammed with these lovely debug messages:
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)From looking at the code, it looks like that each of those messages (there are thousands) equates to 100 trips through the loop. My guess is that the larger blocks require multi-page allocations which are harder to satisfy. This is with 3.10 kernel.
No, larger blocks simply require more single pages. The buffer cache does not require multi-page allocation at all. So, mode = 0x250, which means ___GFP_NOWARN | ___GFP_IO | ___GFP_WAIT which is also known as a GFP_NOFS allocation context.
So, it's entirely possible that your memory is full of cached filesystem data and metadata, and the allocation that needs more can't reclaim them.
Quoted 20 lines
The hardware is something like (I can find out the exact config is you want):
32 cores 128 GB RAM LSI 9271-8i RAID (one big RAID-60 with 36 disks, partitioned)As I hinted at earlier, we end up with pretty big directories. We can semi-reliably trigger this when we run rsync on the data between two (identical) hosts over 10GbitE.
# xfs_info /dev/sda9 meta-data=/dev/sda9 isize=256 agcount=6, agsize=268435455 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1454213211, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=65536 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
Quoted 1 line
/proc/slabinfo: https://www.copy.com/s/1x1yZFjYO2EI/slab.txt
Hmmm. You're using filestreams. That's unusual.
Only major slab cache is the buffer_head slab, with ~12 million active bufferheads. So, that means you've got at least 47-48GB of data in the page cache.....
And there's only ~35000 xfs_buf items in the slab, so the metadata cache isn't very big, and reclaim from that isn't a problem, nor the inode caches as there's only 130,000 cached inodes.
Quoted 1 line
sysrq m output: https://www.copy.com/s/mYfMYfJJl2EB/sysrq-m.txt
27764401 total pagecache pages
which indicates that you've got close to 110GB of pages in the page cache. Hmmm, and 24-25GB of dirty pages in memory.
You know, I'd be suspecting a memory reclaim problem here to do with having large amounts of dirty memory in the page cache. I don't think the underlying cause is going to be the filesystem code, as the warning should never be emitted if memory reclaim is making progress. Perhaps you could try lowering all the dirty memory thresholds to see if that allows memory reclaim to make more progress because there are fewer dirty pages in memory...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Dave Chinner
Quoted 10 lines · Josef 'Jeff' Sipek
We've started experimenting with larger directory block sizes to avoid directory fragmentation. Everything seems to work fine, except that the log is spammed with these lovely debug messages:
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)From looking at the code, it looks like that each of those messages (there are thousands) equates to 100 trips through the loop. My guess is that the larger blocks require multi-page allocations which are harder to satisfy. This is with 3.10 kernel.
Quoted 4 lines
No, larger blocks simply require more single pages. The buffer cache does not require multi-page allocation at all. So, mode =
0x250, which means___GFP_NOWARN|___GFP_IO|___GFP_WAITwhich is also known as aGFP_NOFSallocation context.
Doh! Not sure why I didn't remember the fact that directories are no different from regular files...
...
Quoted 3 lines
Quoted 1 line
/proc/slabinfo: https://www.copy.com/s/1x1yZFjYO2EI/slab.txtHmmm. You're using filestreams. That's unusual.
Right. I keep forgetting about that.
Quoted 7 lines
Quoted 1 line
sysrq m output: https://www.copy.com/s/mYfMYfJJl2EB/sysrq-m.txt
27764401 total pagecache pages
which indicates that you've got close to 110GB of pages in the page cache. Hmmm, and 24-25GB of dirty pages in memory.
Quoted 7 lines
You know, I'd be suspecting a memory reclaim problem here to do with having large amounts of dirty memory in the page cache. I don't think the underlying cause is going to be the filesystem code, as the warning should never be emitted if memory reclaim is making progress. Perhaps you could try lowering all the dirty memory thresholds to see if that allows memory reclaim to make more progress because there are fewer dirty pages in memory...
Yep. This makes perfect sense. Amusingly enough, we don't read much of the data so really the pagecache is supposed to buffer the writes because I/O is slow. We'll play with the dirty memory thresholds and see if that helps.
Thanks!
Jeff.
Signature · 10 lines
All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can’t get them together again, there must be a reason. By all means, do not use a hammer. — IBM Manual, 1925 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs