Latencies in XFS.

Hi,

I'll try not to flood with information on the first post.

In trying to track down this issue here:
http://www.spinics.net/lists/raid/msg17195.html

I think I have narrowed it down to XFS.

The basic problem I am seeing is that applications on client workstations whose home directories are NFS mounted are stalling in filesystem calls such as open, close, unlink.

I have eventually come up with a simple test case which shows the problem (well the seeming filesystem stalls anyway). I did this on my machine at home, Athlon XP 768MB RAM, XFS is on a Seagate IDE. Kernel is 2.8.23-rc9-current git

(Oh yeah, no networking/NFS is involved here)

If I run the following program

/* fslattest.c */

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <string.h>

int main(int argc, char *argv[])

{
        char file[255];

        if (argc < 2) {
                printf("Usage: fslattest file\n");
                exit(1);
        }

        strncpy(file, argv[1], 254);
        printf("Opening %s\n", file);

        while (1) {
                int testfd = open(file, 
                        O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600);
                close(testfd);
                unlink(file);
                sleep(1);
        }

        exit(0);
}

e.g $ strace -T -e open fslattest test

And then after a few seconds run

$ dd if=/dev/zero of=bigfile bs=1M count=500

I see the following

Before dd kicks in

open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.005043>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000212>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.016844>

while dd is running

open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <2.000348>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <1.594441>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <2.224636>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <1.074615>

dd stopped

open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.013224>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.007109>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.007108>

Doing the same thing with ext3 shows no such stalls. e.g before, during and after the above dd

open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.015423>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000092>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000093>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000088>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000103>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000096>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000094>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000114>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000091>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000274>
open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.000107>

I Just tried the above on a machine here in the office. It seems to have a much faster disk than mine, and the latencies aren't quite a dramatic upto about 1.0 seconds. Mounting with nobarrier reduces that to generally < 0.5 seconds.

Just ask if you'd like more info.

Cheers,

Andrew

Le Tue, 9 Oct 2007 16:36:35 +0100 vous écriviez:

I notice you use a bleeding-edge unstable kernel, with a whole new scheduler. I tried your benchmark on a machine running a known stable kernel (2.6.20.17) and the slowdown is similar in xfs and other fs.

As a side note, I personnally wouldn't choose xfs for desktop users home directories (email,web, etc) because it's much better at pure throughput and quite slower at IOPS than other FSes. Reiserfs is the best of the common FSes for this.

I'm quite surprised that apparently your production system is running a prerelease kernel. Why?

--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

I don't see the slowdown in ext3.

Actually it's now running 2.6.23

Le Sun, 14 Oct 2007 01:23:23 +0100 vous écriviez:

I tried it and the slowdown is proportionnally similar, though XFS is slower IO-wise.

--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

andrew> [ ... ] The basic problem I am seeing is that andrew> applications on client workstations whose home andrew> directories are NFS mounted are stalling in filesystem andrew> calls such as open, close, unlink.

Metadata and data sync/flushing can be handled very differently by the same filesystem and by different filesystems.

[ ... ]

andrew> [ ... ] e.g $ strace -T -e open fslattest test andrew> And then after a few seconds run andrew> $ dd if=/dev/zero of=bigfile bs=1M count=500 andrew> I see the following

So metadata and data sync/flush are competing, and also 'dd' is hitting heavily the buffer cache, with 500MB on a 768MB system.

andrew> Before dd kicks in
andrew> open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <0.005043> andrew> [ ... ] while dd is running
andrew> open("test", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0600) = 3 <2.000348> andrew> [ ... ] Doing the same thing with ext3 shows no such
andrew> stalls.

All this does not sound that susprising to me.

[ ... ]

andrew> I Just tried the above on a machine here in the andrew> office. It seems to have a much faster disk than mine, andrew> and the latencies aren't quite a dramatic upto about 1.0 andrew> seconds. Mounting with nobarrier reduces that to andrew> generally < 0.5 seconds.

That's a rather clear hint I suppose.

My suggestion would be to run 'vmstat 1' watching in particular the cache and 'bi'/'bo' columns while doing experiments with:

* Increasing value of the 'commit' mount option of 'ext3'.
* Different value of the 'data' mount option of 'ext3'.
* The elevator algorithm for the affected disks.
* Changing values of '/proc/sys/net/bdflush'.
* The 'oflag=direct' option of 'dd'.

And the impact the above have on the memory-write-caching of XFS and the ordering of CPU and disk operations in general.

There are many odd interactions of these parameters, here is an example of a discussion of a different case:

http://www.sabi.co.uk/blog/0707jul.html#070701b

Also, having a look at some bits of your Linux RAID list post:

It might we useful to reconsider some of the build vs. mount parameters, and the chunk size (consider the non trivial issue of excessive stripe length).