Skip to contentSkip to conversation

xfstests failure generic/239

DISCUSSION Zhao Hongjiang 3 participants 16 messages Jun 8, 2013 – Aug 1, 2013

lore
Share & tools
lore
#0 lore raw

Hi all,

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0
     ...

I found that from 2.6.39 the test case occur this failure and it is is introduced by this
commit: 198868f35de99e7197829314076e5465c37e4cc5
ext4: Use single thread to perform DIO unwritten convertion

When i revert this patch, the failure seems go away. But when i see
http://oss.sgi.com/archives/xfs/2010-06/msg00269.html, it said that
"This contains core direct I/O changes to allow the filesystem to delay AIO completion, as well as a patch to fix XFS. ext4 also has the same issue, and from a quick look also doesn't properly complete unwritten extent conversions for synchronous direct I/O, but I'll leave that for someone more familar to figure out."

I also run the test against 3.4.24 rt-tree with revert the patch which i found before, but the hundred times tests all failure.

So, things maybe not simple, i doubt the patch i found isn't the root cause.

Regards,
- Hongjiang

#1 lore raw
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

#2 lore raw
Quoted 40 lines · Theodore Ts'o
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Quoted 5 lines

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure?

I'am use x86 and start a qumu-kvm to run the test.

Quoted 2 lines

I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps

And as Christoph Hellwig said "the race is very easy to hit by using QEMU with native AIO support on a sparse image, and the result is filesystem corruption in the guest", i also run the test on the host, but nerver see the failure.

Quoted 3 lines

wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

This seems necessary, i'll give a patch for this.
- Zhao

Quoted 9 lines

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#3 lore raw
Quoted 47 lines · Zhao Hongjiang
Quoted 40 lines · Theodore Ts'o
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Quoted 5 lines

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure?

Quoted 8 lines

I'am use x86 and start a qumu-kvm to run the test.

Quoted 2 lines

I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps

And as Christoph Hellwig said "the race is very easy to hit by using QEMU with native AIO support on a sparse image, and the result is filesystem corruption in the guest", i also run the test on the host, but nerver see the failure.

Sorry, i run the test on host a hundred times again, there are six failures.

Quoted 17 lines
Quoted 4 lines

wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

This seems necessary, i'll give a patch for this.
- Zhao

Quoted 9 lines

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#4 lore raw
Quoted 42 lines · Zhao Hongjiang
Quoted 41 lines · Zhao Hongjiang
Quoted 40 lines · Theodore Ts'o
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure?

I'am use x86 and start a qumu-kvm to run the test.

more informations:

8 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

Host: scsi0 Channel: 02 Id: 01 Lun: 00
Vendor: LSI Model: MegaRAID SAS RMB Rev: 1.40
Type: Direct-Access ANSI SCSI revision: 05

Quoted 36 lines
Quoted 7 lines
Quoted 3 lines

I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps

And as Christoph Hellwig said "the race is very easy to hit by using QEMU with native AIO support on a sparse image, and the result is filesystem corruption in the guest", i also run the test on the host, but nerver see the failure.

Sorry, i run the test on host a hundred times again, there are six failures.

Quoted 17 lines
Quoted 4 lines

wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

This seems necessary, i'll give a patch for this.
- Zhao

Quoted 9 lines

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

.

#5 lore raw
In reply to Theodore Ts'o

Hi, Ted

How about this problem? Is there some progress?

Hongjiang

Quoted 59 lines · Theodore Ts'o
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#6 lore raw
In reply to Theodore Ts'o

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch:
commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4: use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Regards,
Zhao

Quoted 59 lines · Theodore Ts'o
Quoted 7 lines · Zhao Hongjiang

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#7 lore raw
Quoted 6 lines · Zhao Hongjiang

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Honza

Quoted 62 lines · Theodore Ts'o
Quoted 59 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#8 lore raw
Quoted 16 lines · Jan Kara
Quoted 6 lines · Zhao Hongjiang

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,

        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);

+ ext4_io_end_t *io_end = NULL;

        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;

+ }
/*
* The io_end structure takes a reference to the inode,
* that structure needs to be destroyed and the

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

Regards,
Zhao

Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#9 lore raw
Quoted 60 lines · Zhao Hongjiang
Quoted 16 lines · Jan Kara
Quoted 6 lines · Zhao Hongjiang

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 62 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#10 lore raw

Hi, Jan

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

Regards,
Zhao

Quoted 132 lines · Jan Kara
Quoted 60 lines · Zhao Hongjiang
Quoted 16 lines · Jan Kara
Quoted 6 lines · Zhao Hongjiang

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#11 lore raw

Hi,

Quoted 2 lines · Zhao Hongjiang

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

OK, so the test is still failing after using io_end instead of
iocb->private? If yes, I'm not sure where the problem exactly is, sorry.

Honza

Quoted 135 lines · Jan Kara
Quoted 132 lines · Zhao Hongjiang
Quoted 60 lines · Jan Kara
Quoted 16 lines · Zhao Hongjiang
Quoted 6 lines

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#12 lore raw
Quoted 7 lines · Jan Kara

Hi,

Quoted 2 lines · Zhao Hongjiang

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

OK, so the test is still failing after using io_end instead of
iocb->private? If yes, I'm not sure where the problem exactly is, sorry.

Just one addition: But from the fact that debug printk triggers we see
that really the problem happens when DIO is completed before __blockdev_direct_IO() returns and thus extent conversion can race with parts of __blockdev_direct_IO() or ext4_ext_direct_IO(). That is impossible after my patch because ext4_ext_direct_IO() holds a reference to io_end so it gets queued for completion only after __blockdev_direct_IO() returns. So at least I somewhat understand why my patch makes a difference.

Honza

Quoted 138 lines
Quoted 135 lines · Jan Kara
Quoted 132 lines · Zhao Hongjiang
Quoted 60 lines · Jan Kara
Quoted 16 lines · Zhao Hongjiang
Quoted 6 lines

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#13 lore raw
Quoted 8 lines · Jan Kara
Quoted 7 lines · Jan Kara

Hi,

Quoted 2 lines · Zhao Hongjiang

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

OK, so the test is still failing after using io_end instead of
iocb->private? If yes, I'm not sure where the problem exactly is, sorry.

Quoted 8 lines

Just one addition: But from the fact that debug printk triggers we see
that really the problem happens when DIO is completed before __blockdev_direct_IO() returns and thus extent conversion can race with parts of __blockdev_direct_IO() or ext4_ext_direct_IO(). That is impossible after my patch because ext4_ext_direct_IO() holds a reference to io_end so it gets queued for completion only after __blockdev_direct_IO() returns. So at least I somewhat understand why my patch makes a difference.

So is this might be the solution to fix the problem?

Regards,
Zhao

Quoted 139 lines
Quoted 139 lines
Quoted 136 lines · Jan Kara
Quoted 132 lines · Zhao Hongjiang
Quoted 60 lines · Jan Kara
Quoted 16 lines · Zhao Hongjiang
Quoted 6 lines

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#14 lore raw
In reply to Jan Kara
Quoted 8 lines · Jan Kara

Hi,

Quoted 2 lines · Zhao Hongjiang

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

OK, so the test is still failing after using io_end instead of
iocb->private? If yes, I'm not sure where the problem exactly is, sorry.

I hit the bug just with the follow code that you give out:

	if (io_end != NULL) {

if (iocb->private == NULL)
printk("Bug happened!\n");
EXT4_I(inode)->cur_aio_dio = NULL;
}

With this the "Bug happened!" is come out everytime while the test is fail. But if the test case is pass, the "Bug happened!" never come out!

Quoted 136 lines
Quoted 136 lines · Jan Kara
Quoted 132 lines · Zhao Hongjiang
Quoted 60 lines · Jan Kara
Quoted 16 lines · Zhao Hongjiang
Quoted 6 lines

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

#15 lore raw
Quoted 17 lines · Zhao Hongjiang
Quoted 8 lines · Jan Kara

Hi,

Quoted 2 lines · Zhao Hongjiang

It hit this bug, the "Bug happened!" is come out everytime while the test is fail. Any suggestion for fix this?

OK, so the test is still failing after using io_end instead of
iocb->private? If yes, I'm not sure where the problem exactly is, sorry.

I hit the bug just with the follow code that you give out:

	if (io_end != NULL) {
(quoted diff) +0 -0
 		if (iocb->private == NULL)
 			printk("Bug happened!\n");
 		EXT4_I(inode)->cur_aio_dio = NULL;
 	}
Quoted 2 lines

With this the "Bug happened!" is come out everytime while the test is fail. But if the test case is pass, the "Bug happened!" never come out!

Yes, so we know early completion of direct IO triggers the bug. The above
condition fixes a bug in ext4_ext_direct_IO() but as you are still getting test failures, there must be some other bug still present. And I don't know where it is...

Honza

Quoted 137 lines
Quoted 135 lines
Quoted 135 lines · Jan Kara
Quoted 132 lines · Zhao Hongjiang
Quoted 60 lines · Jan Kara
Quoted 16 lines · Zhao Hongjiang
Quoted 6 lines

Hi, jack

I test the latest kernel 3.11-rc2 and it seems the problem is fix by the follow patch: commit id:97a851ed71cd9cc2542955e92a001c6ea3d21d35 (ext4:
use io_end for multiple bios). But it's so difficult to backport to kernel 3.4-stable, any suggestion for this?

Backporting that patch to stable kernels is no-go. It is far to intrusive
for stable kernels. I was looking for a while how that patch could fix the problem you were observing. I think there is a subtle race possible when AIO DIO write completes before __blockdev_direct_IO() returns. In that case we set iocb->private to NULL in ext4_end_io_dio() but we also key off iocb->private in ext4_ext_direct_IO() as:

                if (iocb->private)
                        ext4_inode_aio_set(inode, NULL);
Quoted 9 lines

So in the case above we forget to reset inode's AIO pointer. That can then cause strange effects with unwritten extent handling (although I admit I'm not sure whether it can also cause the failure you observe) and 97a851ed71cd9cc2542955e92a001c6ea3d21d35 actually fixes that bug. You can easily check whether you are hitting that bug or not by changing the above condition from testing iocb->private to testing some private variable... E.g. you could declare io_end and set it to NULL one level up in ext4_ext_direct_IO() and then test io_end != NULL in that condition.

Thanks for your reply first.
I change the code like the follow:

(quoted diff) +5 -3
@@ -2921,6 +2921,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
        struct inode *inode = file->f_mapping->host;
        ssize_t ret;
        size_t count = iov_length(iov, nr_segs);
+       ext4_io_end_t *io_end = NULL;
 
        loff_t final_size = offset + count;
        if (rw == WRITE && final_size <= inode->i_size) {
@@ -2947,8 +2948,7 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                iocb->private = NULL;
                EXT4_I(inode)->cur_aio_dio = NULL;
                if (!is_sync_kiocb(iocb)) {
-                       ext4_io_end_t *io_end =
-                               ext4_init_io_end(inode, GFP_NOFS);
+                       io_end = ext4_init_io_end(inode, GFP_NOFS);
                        if (!io_end)
                                return -ENOMEM;
                        io_end->flag |= EXT4_IO_END_DIRECT;
@@ -2970,8 +2970,10 @@ static ssize_t ext4_ext_direct_IO(int rw, struct kiocb *iocb,
                                         ext4_end_io_dio,
                                         NULL,
                                         DIO_LOCKING);
-               if (iocb->private)
+               if (io_end != NULL) {
+                       printk("Zhao Hongjiang Ext4 test!\n");
                        EXT4_I(inode)->cur_aio_dio = NULL;
+               }
                /*
                 * The io_end structure takes a reference to the inode,
                 * that structure needs to be destroyed and the
Quoted 2 lines

And the print come out when i run the test everytime. So i think the test hit the bug that you mentioned, Am i right or miss something?

It is not a bug that you hit the branch with printk(). It would be a bug
if the debug check looked like:

	if (io_end != NULL) {
		if (iocb->private == NULL)
			printk("Bug happened!\n");
		EXT4_I(inode)->cur_aio_dio = NULL;
	}

								Honza
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines · Theodore Ts'o
Quoted 58 lines · Zhao Hongjiang
Quoted 7 lines

I run xfstests #239 against mainline 3.10.0-rc3, unfortunately it failure in my QEMU. I run the case a hundred times, it certainly hit the failure several times. The failure msg is as follow:

FSTYP -- ext4
PLATFORM -- Linux/x86_64 3.10.0-rc3-mainline

Quoted 6 lines

generic/239 1s ... - output mismatch (see /home/zhj/xfstests/results/generic/239.out.bad)

    --- tests/generic/239.out   2013-06-07 22:04:09.000000000 -0400
    +++ /home/zff/xfstests/results/generic/239.out.bad  2013-06-07 22:04:09.000000000 -0400
    @@ -1,2 +1,515 @@
     QA output created by 239
    +hostname: Host name lookup failure

OK, so this hostname failure is weird; I'm not sure what's causing this, but this I presume unrelated to the failure at hand.

Quoted 5 lines
     Silence is golden
    +0: 0x0
    +1: 0x0
    +2: 0x0
    +3: 0x0

This indicates a problem. Test generic/239 is running aio-dio-hole-filling-race.c, which submits an asynchronous, direct I/O 4k write with a buffer containing non-zero contents to a sparse file, and once the I/O has completed, it uses pread to read it back, using the same descriptor, so it is doing the read using direct I/O. It then checks to see if the read returns zero or not.

The "XX: 0x0" lines indicates that buffer is zero, which implies that somehow aio_complete() is getting called before the uninitialized to initialized conversion is taking place. I'm not seeing how this is happening, though, so I'm a bit puzzled. If there are any unwritten extents, we don't call aio_complete() in ext4_end_io_dio(), but instead the conversion is queued via a call to ext4_add_compete_io(), and and aio_done() is only called on the iocb after the conversion is complete.

Can anyone see something that I might be missing?

- Ted

P.S. Zhao, what was the hardware that you using to find this failure? I'm not seeing it, but then again if the failure is only happening once every few hundred runs that might explain it. I'm perhaps wondering if we should add a mode to aio-dio-hole-filling-race.c which allows it to try the race a large number of times, instead of just once.

P.P.S. One thought.... perhaps it might be useful to have a debug mode where we use queue_delayed_work() to submit the conversion request to the workqueue. It will of course make certain workloads run slow as molasses, but it might expose some races so we can see them more easily.

.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR