Tweak tracing allocation sizes
PATCH 1/1 Tweak tracing allocation sizes
The size of a single ktrace entry is 16 pointers so 128 bytes. For the case of XFS_RW_KTRACE_SIZE which is 128 entries this equates to 16KB and on a system with 4KB pages that is under memory pressure this can stall that process for a significant time while it hunts for 4 free pages. Cutting this value back to 32 means it will only need one page.
Initialize the ktrace system with a zone size of 32 so calls to ktrace_alloc() that want 32 trace entries (ie 1 page) will go through the ktrace entry zone. Bump INODE_TRACE_SIZE from 16 to 32 since if we are going to allocate half a page we might as well give it a full page and have it allocate from the zone too.
More can be done here but these changes reduce the liklihood of hitting deadlocks due to memory pressure.
fs/xfs/linux-2.6/xfs_lrw.h +0 -0
@@ -28,7 +28,7 @@ struct xfs_iomap;
/*
* Defines for the trace mechanisms in xfs_lrw.c.
*/
-#define XFS_RW_KTRACE_SIZE 128
+#define XFS_RW_KTRACE_SIZE 32
#define XFS_READ_ENTER 1
#define XFS_WRITE_ENTER 2
fs/xfs/linux-2.6/xfs_super.c +0 -0
@@ -2151,7 +2151,7 @@ init_xfs_fs(void)
printk(message);
- ktrace_init(64);
+ ktrace_init(32); vn_init(); xfs_dir_startup();
fs/xfs/linux-2.6/xfs_vnode.h +0 -0
@@ -126,7 +126,7 @@ static inline void vn_atime_to_time_t(st*/
#if defined(XFS_INODE_TRACE)
-#define INODE_TRACE_SIZE 16 /* number of trace entries */
+#define INODE_TRACE_SIZE 32 /* number of trace entries */
#define INODE_KTRACE_ENTRY 1
#define INODE_KTRACE_EXIT 2
#define INODE_KTRACE_HOLD 3
Quoted 5 lines · Lachlan McIlroy
The size of a single ktrace entry is 16 pointers so 128 bytes. For the case of
XFS_RW_KTRACE_SIZEwhich is 128 entries this equates to 16KB and on a system with 4KB pages that is under memory pressure this can stall that process for a significant time while it hunts for 4 free pages. Cutting this value back to 32 means it will only need one page.
That will effectively render that type of tracing useless - 32 entries is not enough history to capture enough read/write/map/invalidate trace events to be meaningful. In the past I've often had to increase this to 256 or 512 entries to be able to capture the events necessary to debug problems...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 7 lines · Dave Chinner
Quoted 5 lines · Lachlan McIlroy
The size of a single ktrace entry is 16 pointers so 128 bytes. For the case of
XFS_RW_KTRACE_SIZEwhich is 128 entries this equates to 16KB and on a system with 4KB pages that is under memory pressure this can stall that process for a significant time while it hunts for 4 free pages. Cutting this value back to 32 means it will only need one page.
Quoted 5 lines
That will effectively render that type of tracing useless - 32 entries is not enough history to capture enough read/write/map/invalidate trace events to be meaningful. In the past I've often had to increase this to 256 or 512 entries to be able to capture the events necessary to debug problems...
A system that constantly locks up and/or stalls is useless too. Allocating 4 or more pages for every inode just taxes the system. Can you offer an alternative - maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
Quoted 14 lines · Lachlan McIlroy
Quoted 7 lines · Dave Chinner
Quoted 5 lines · Lachlan McIlroy
The size of a single ktrace entry is 16 pointers so 128 bytes. For the case of
XFS_RW_KTRACE_SIZEwhich is 128 entries this equates to 16KB and on a system with 4KB pages that is under memory pressure this can stall that process for a significant time while it hunts for 4 free pages. Cutting this value back to 32 means it will only need one page.Quoted 5 lines
That will effectively render that type of tracing useless - 32 entries is not enough history to capture enough read/write/map/invalidate trace events to be meaningful. In the past I've often had to increase this to 256 or 512 entries to be able to capture the events necessary to debug problems...
Quoted 3 lines
A system that constantly locks up and/or stalls is useless too. Allocating 4 or more pages for every inode just taxes the system. Can you offer an alternative
Buy more memory for your test machine? (I can't beleive I'm saying that to an SGI guy ;)
How about using the SLUB allocator rather than SLAB and tweaking it's settings to do order 2 allocations for every slab so most allocations are doing order 2 allocations?
Quoted 2 lines
- maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
Sure, we've got that for various other trace types (e.g. the "vnode" trace). You'd need to add idbg stuff for filtering based on the inode the buffer belongs to....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 6 lines · Dave Chinner
Quoted 2 lines
- maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
Sure, we've got that for various other trace types (e.g. the "vnode" trace). You'd need to add idbg stuff for filtering based on the inode the buffer belongs to....
That's probably the best idea. Currently all these per-object ktrace buffers do really large no-MAYFAIL allocation all over. They are in fact the remaining reason not to simply directly call vmalloc for large allocations instead of all our current mess in the kmem_ functions.
Quoted 13 lines · Lachlan McIlroy
Quoted 11 lines · Dave Chinner
Quoted 5 lines · Lachlan McIlroy
The size of a single ktrace entry is 16 pointers so 128 bytes. For the case of
XFS_RW_KTRACE_SIZEwhich is 128 entries this equates to 16KB and on a system with 4KB pages that is under memory pressure this can stall that process for a significant time while it hunts for 4 free pages. Cutting this value back to 32 means it will only need one page.That will effectively render that type of tracing useless - 32 entries is not enough history to capture enough read/write/map/invalidate trace events to be meaningful. In the past I've often had to increase this to 256 or 512 entries to be able to capture the events necessary to debug problems...
Quoted 4 lines
A system that constantly locks up and/or stalls is useless too. Allocating 4 or more pages for every inode just taxes the system. Can you offer an alternative - maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
You could use vmalloc(). While that is also not fast it will at least not stall.
-Andi
Signature · 1 line
[email protected]
Quoted 5 lines · Andi Kleen
Quoted 2 lines
alternative - maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
You could use
vmalloc(). While that is also not fast it will at least not stall.
In fact kmem_alloc first tries vmalloc, and then falls back to slab when it fails. See fs/xfs/linux-2.6/kmem.c:kmem_alloc().
Yes, it's all a big mess..
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Andi Kleen
Quoted 2 lines
alternative - maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
You could use
vmalloc(). While that is also not fast it will at least not stall.In fact
kmem_allocfirst tries vmalloc, and then falls back to slab when it fails. See fs/xfs/linux-2.6/kmem.c:kmem_alloc().
You mean the other way around?
Anyways it sounds like like MAX_SLAB_SIZE is just too big.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Andi Kleen
Quoted 2 lines
alternative - maybe a very large global trace buffer that is allocated at mount time and shared by all inodes?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
You could use
vmalloc(). While that is also not fast it will at least not stall.In fact
kmem_allocfirst tries vmalloc, and then falls back to slab when it fails. See fs/xfs/linux-2.6/kmem.c:kmem_alloc().You mean the other way around?
No, take a look at the function. I guess the intent is that vmalloc can fail due to a full vmalloc area and kmalloc could theoretically still scucceed.