Skip to contentSkip to conversation

xfsaild causing 30+ wakeups/s on an idle system since 2.6.25-rcX

DISCUSSION Török Edwin 4 participants 7 messages Feb 17, 2008 – Feb 19, 2008 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi,

xfsaild is causing many wakeups, a quick investigation shows xfsaild_push is always
returning 30 msecs timeout value.

This is on an idle system, running only gnome, and gnome-terminal.

I suggest changing the timeout logic in xfsaild to be more power consumption friendly.

See below my original report to the powerTOP mailing list.

Best regards,
--Edwin

Quoted 14 lines · Török Edwin
Quoted 5 lines · Török Edwin

Hi,

On latest -git of 2.6.25 I am getting lots of wakeups from xfsaild.
23.5% ( 33.3) xfsaild : schedule_timeout (process_timeout)

[Should I Cc: xfs mailing list / lkml on this?]

The problem seems to be with the timeout logic in xfsaild_push, which can return 3 timeout values (msecs): 1000, 10, 20, 30.
I inserted a marker and attached a probe function, schedule_timeout always got called with 9 jiffies (which is 30 msecs, I have HZ=300).

Quoted 6 lines

Changing xfs_trans_ail.c:270 from "tout += 20" to "tout = 1000", made xfsaild only do 1 wakeup/s instead of 33!

For some reason xfs_aild always thinks it has work (I/O) to do, and never chooses the 1000 msec sleep value.

sounds like an XFS bug... worth reporting to the xfs/lkml folks for sure.

#1 lore raw
Quoted 16 lines · Török Edwin

Hi,

xfsaild is causing many wakeups, a quick investigation shows xfsaild_push is always
returning 30 msecs timeout value.

This is on an idle system, running only gnome, and gnome-terminal.

I suggest changing the timeout logic in xfsaild to be more power consumption friendly.

See below my original report to the powerTOP mailing list.

Best regards,
--Edwin

Signature · 2 lines
Thanks,
Oliver
#2 lore raw
Quoted 6 lines · Oliver Pinter
Quoted 5 lines · Török Edwin

Hi,

xfsaild is causing many wakeups, a quick investigation shows xfsaild_push is always
returning 30 msecs timeout value.

That's a bug, and has nothing to do with power consumption. ;)

I can see that there is a dirty item in the filesystem:

Entering kdb (current=0xe00000b8f4fe0000, pid 30046) on processor 3 due to Breakpoint @ 0xa000000100454fc0 [3]kdb> bt
Stack traceback for pid 30046
0xe00000b8f4fe0000 30046 2 1 3 R 0xe00000b8f4fe0340 *xfsaild
0xa000000100454fc0 xfsaild_push

        args (0xe00000b8ffff9090, 0xe00000b8f4fe7e30, 0x31b)

....
[3]kdb> xail 0xe00000b8ffff9090
AIL for mp 0xe00000b8ffff9090, oldest first
[0] type buf flags: 0x1 <in ail > lsn [1:13133]
buf 0xe00000b880258800 blkno 0x0 flags: 0x2 <dirty > Superblock (at 0xe00000b8f9b3c000)
[3]kdb>

the superblock is dirty, and the lsn is well beyond the target of the xfsaild hence it *should* be idling. However, it isn't idling because there is a dirty item in the list and the idle trigger of "is list empty" is not tripping.

I only managed to reproduce this on a lazy superblock counter filesystem (i.e. new mkfs and recent kernel), as it logs the superblock every so often, and that is probably what is keeping the fs dirty like this.

Can you see if the patch below fixes the problem.

---

Idle state is not being detected properly by the xfsaild push code. The current idle state is detected by an empty list which may never happen with mostly idle filesystem or one using lazy superblock counters. A single dirty item in the list will result repeated looping to push everything past the target when everything because it fails to check if we managed to push anything.

Fix by considering a dirty list with everything past the target as an idle state and set the timeout appropriately.

---
 fs/xfs/xfs_trans_ail.c |   15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
fs/xfs/xfs_trans_ail.c +9 -6
Index: 2.6.x-xfs-new/fs/xfs/xfs_trans_ail.c
===================================================================
@@ -261,14 +261,17 @@ xfsaild_push(
 		xfs_log_force(mp, (xfs_lsn_t)0, XFS_LOG_FORCE);
 	}
 
-	/*
-	 * We reached the target so wait a bit longer for I/O to complete and
-	 * remove pushed items from the AIL before we start the next scan from
-	 * the start of the AIL.
-	 */
-	if ((XFS_LSN_CMP(lsn, target) >= 0)) {
+	if (count && (XFS_LSN_CMP(lsn, target) >= 0)) {
+		/*
+		 * We reached the target so wait a bit longer for I/O to
+		 * complete and remove pushed items from the AIL before we
+		 * start the next scan from the start of the AIL.
+		 */
 		tout += 20;
 		last_pushed_lsn = 0;
+	} else if (!count) {
+		/* We're past our target or empty, so idle */
+		tout = 1000;
 	} else if ((restarts > XFS_TRANS_PUSH_AIL_RESTARTS) ||
 		   (count && ((stuck * 100) / count > 90))) {
 		/*
#3 lore raw
Quoted 4 lines · David Chinner
Quoted 2 lines · Oliver Pinter

On 2/17/08, Török Edwin <[email protected]> wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Hi,

xfsaild is causing many wakeups, a quick investigation shows xfsaild_push is always
returning 30 msecs timeout value.

That's a bug

Ok. Your patches fixes the 30+ wakeups :)

Quoted 2 lines

, and has nothing to do with power consumption. ;)

I suggest using a sysctl value (such as /proc/sys/vm/dirty_writeback_centisecs), instead of a hardcoded default 1000.
That would further reduce the wakeups.

Quoted 3 lines

I only managed to reproduce this on a lazy superblock counter filesystem (i.e. new mkfs and recent kernel),

The filesystem was created in July 2007

Quoted 1 line

Can you see if the patch below fixes the problem.

Yes, it reduces wakeups to 1/sec.

Thanks,
--Edwin

#4 lore raw
Quoted 7 lines · Török Edwin
Quoted 6 lines · David Chinner
Quoted 4 lines · Oliver Pinter
Quoted 2 lines · Török Edwin

Hi,

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

xfsaild is causing many wakeups, a quick investigation shows xfsaild_push is always
returning 30 msecs timeout value.

That's a bug

Ok. Your patches fixes the 30+ wakeups :)

Good. I'll push it out for review then.

Quoted 6 lines
Quoted 2 lines

, and has nothing to do with power consumption. ;)

I suggest using a sysctl value (such as /proc/sys/vm/dirty_writeback_centisecs), instead of a hardcoded default 1000.

No, too magic. I dislike adding knobs to workaround issues that really should be fixed by having sane default behaviour. Further down the track as we correct know issues with the AIL push code we'll be able to increase this idle timeout or even make it purely wakeup driven once we get back to an idle state. However, right now it still needs that once a second wakeup to work around a nasty corner case that can hang the filesystem....

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#5 lore raw
In reply to Török Edwin

Not to look excessively dumb, but what's xfsaild?

xfs seems to be sprouting daemons at a more rapid pace these days...xfsbufd, xfssyncd, xfsdatad, xfslogd, xfs_mru_cache, and now xfsaild?

Not a complaint if it ups performance, but I do sorta wonder what all of them are for and why they are needed "now" but not for, say, kernels before 2.6.18 (arbitrary number picked out of hat).

Like bufd writes out buffers, logd writes/hands the log, datad? Isn't the data in buffers? mru_cache? -- isn't that handled by the linux block layer? Sorry...just a bit confused by the additions...

Are there any design docs (scribbles?) saying what these do and why they were added so I can just go read 'em myself? I'm sure they were added for good reason...just am curious more than anything.

Thanksd
-lindad

#6 lore raw
Quoted 1 line · Linda Walsh

Not to look excessively dumb, but what's xfsaild?

AIL = Active Item List

It is a sorted list all the logged metadata objects that have not yet been written back to disk. The xfsaild is responsible for tail pushing the log. i.e. writing back objects in the AIL in the most efficient manner possible.

Why a thread? Because allowing parallelism in tail pushing is a scalability problem and moving this to it's own thread completely isolates it from parallelism. Tail pushing only requires a small amount of CPU time, but it requires a global scope spinlock. Isolating the pushing to a single CPU means the spinlock is not contended across every CPU in the machine.

How much did it improve scalability? on a 2048p machine with an MPI job that did a synchronised close of 12,000 files (6 per CPU), the close time went from ~5400s without the thread to 9s with the xfsaild. That's only about 600x faster. ;)

Quoted 3 lines

xfs seems to be sprouting daemons at a more rapid pace these days...xfsbufd, xfssyncd, xfsdatad, xfslogd, xfs_mru_cache, and now xfsaild?

Why not? Got to make use of all those cores machines have these days. ;)

Fundamentally, threads are cheap and simple. We'll keep adding threads where it makes sense as long as it improves performance and scalability.

Quoted 3 lines

Are there any design docs (scribbles?) saying what these do and why they were added so I can just go read 'em myself? I'm sure they were added for good reason...just am curious more than anything.

'git log' is your friend. The commits that introduce the new threads explain why they are necessary. ;)

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group