Skip to contentSkip to conversation

Query about DIO/AIO WRITE throttling and ext4 serialization

DISCUSSION Vivek Goyal 5 participants 17 messages Jun 1, 2011 – Jun 9, 2011 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

I know that for throttling of buffered WRITES I do run into serialization issues hence I was thinking of moving throttling buffered WRITE while they are entering page cache and get rid of issues related to filesystem serialization. I am not sure about DIO/AIO WRITES hence the question.

Thanks
Vivek

#1 lore raw
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Quoted 5 lines

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Dave,

Does this serialization happens against that particular inode on which truncate has been called? If yes, then I think I will still be fine as in common use case I am not expecting much sharing of inodes across cgroups.

Thanks
Vivek

#3 lore raw
Quoted 15 lines · Vivek Goyal
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Quoted 5 lines

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Quoted 6 lines

Dave,

Does this serialization happens against that particular inode on which truncate has been called? If yes, then I think I will still be fine as in common use case I am not expecting much sharing of inodes across cgroups.

Dave,

I did a quick test of throttling a direct IO on one file and then doing "truncate -s 40 testfile" on a different file in different cgroup and it seems to work fine.

But I seem to be having issues with "sync". Looks like in ext4, if I throttle a DIO, sync does not hang but in XFS it does. I am wondering if XFS is waiting for all inflight DIO to finish before sync completes.

Thanks
Vivek

#4 lore raw
Quoted 25 lines · Vivek Goyal
Quoted 15 lines · Vivek Goyal
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Quoted 5 lines

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Quoted 6 lines

Dave,

Does this serialization happens against that particular inode on which truncate has been called? If yes, then I think I will still be fine as in common use case I am not expecting much sharing of inodes across cgroups.

Dave,

Quoted 8 lines

I did a quick test of throttling a direct IO on one file and then doing "truncate -s 40 testfile" on a different file in different cgroup and it seems to work fine.

But I seem to be having issues with "sync". Looks like in ext4, if I throttle a DIO, sync does not hang but in XFS it does. I am wondering if XFS is waiting for all inflight DIO to finish before sync completes.

"sync" on XFS seems to be livelocking as long as DIO write operation is going on and same does not happen on ext4.

I ran "aio-stress -O aiofile1 -s 4G" and in other window I did "sync" and it does not finish untile and unless aio-stress has finished. On the other hand ext4 seems to be fine and it does finish earlier.

Thanks
Vivek

#5 lore raw
Quoted 35 lines · Vivek Goyal
Quoted 25 lines · Vivek Goyal
Quoted 15 lines · Vivek Goyal
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Quoted 5 lines

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Quoted 6 lines

Dave,

Does this serialization happens against that particular inode on which truncate has been called? If yes, then I think I will still be fine as in common use case I am not expecting much sharing of inodes across cgroups.

Dave,

Quoted 8 lines

I did a quick test of throttling a direct IO on one file and then doing "truncate -s 40 testfile" on a different file in different cgroup and it seems to work fine.

But I seem to be having issues with "sync". Looks like in ext4, if I throttle a DIO, sync does not hang but in XFS it does. I am wondering if XFS is waiting for all inflight DIO to finish before sync completes.

Quoted 6 lines

"sync" on XFS seems to be livelocking as long as DIO write operation is going on and same does not happen on ext4.

I ran "aio-stress -O aiofile1 -s 4G" and in other window I did "sync" and it does not finish untile and unless aio-stress has finished. On the other hand ext4 seems to be fine and it does finish earlier.

On XFS sync waits for the IO count on each inode to return to zero before continuing. If you are blasting concurrent AIO/DIO at a file, then it is possible that the IO count never falls to zero. It's questionable whether this is necessary, but ISTR that the current behaviour has been there for a long time (though morphed about a bit in implementation).

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#6 lore raw

On Fri, Jun 03, 2011 at 09:51:53AM +1000, Dave Chinner wrote:

[..]

Quoted 18 lines
Quoted 11 lines
Quoted 10 lines

Dave,

I did a quick test of throttling a direct IO on one file and then doing "truncate -s 40 testfile" on a different file in different cgroup and it seems to work fine.

But I seem to be having issues with "sync". Looks like in ext4, if I throttle a DIO, sync does not hang but in XFS it does. I am wondering if XFS is waiting for all inflight DIO to finish before sync completes.

Quoted 6 lines

"sync" on XFS seems to be livelocking as long as DIO write operation is going on and same does not happen on ext4.

I ran "aio-stress -O aiofile1 -s 4G" and in other window I did "sync" and it does not finish untile and unless aio-stress has finished. On the other hand ext4 seems to be fine and it does finish earlier.

Quoted 6 lines

On XFS sync waits for the IO count on each inode to return to zero before continuing. If you are blasting concurrent AIO/DIO at a file, then it is possible that the IO count never falls to zero. It's questionable whether this is necessary, but ISTR that the current behaviour has been there for a long time (though morphed about a bit in implementation).

In this case only a single thread is doing IO continuously. I am assuming if there is a database using XFS, it is not unreasonable to have prolonged periods of continuous IO activity. In that case I think by above design sync will not finish until and unless there is a momentary pause in IO. This does not sound like the best design choice.

Thanks
Vivek

#7 lore raw
Quoted 6 lines · Vivek Goyal

In this case only a single thread is doing IO continuously. I am assuming if there is a database using XFS, it is not unreasonable to have prolonged periods of continuous IO activity. In that case I think by above design sync will not finish until and unless there is a momentary pause in IO. This does not sound like the best design choice.

Sure, but under what circumstances would a database be blasting data using AIO/DIO in one thread, and calling fsync() in another thread? In practice I don't think this situation should ever arise. If it did, the question of which writes could be considered safely on stable store and which would not would be undefined. In fact, for most enterpise databases, they are using preallocated files, so there's no need at all to use fsync() and AIO/DIO at the same time.

- Ted

#8 lore raw
Quoted 8 lines · Ted Ts'o
Quoted 6 lines · Vivek Goyal

In this case only a single thread is doing IO continuously. I am assuming if there is a database using XFS, it is not unreasonable to have prolonged periods of continuous IO activity. In that case I think by above design sync will not finish until and unless there is a momentary pause in IO. This does not sound like the best design choice.

Quoted 7 lines

Sure, but under what circumstances would a database be blasting data using AIO/DIO in one thread, and calling fsync() in another thread? In practice I don't think this situation should ever arise. If it did, the question of which writes could be considered safely on stable store and which would not would be undefined. In fact, for most enterpise databases, they are using preallocated files, so there's no need at all to use fsync() and AIO/DIO at the same time.

In this case I had done "sync" while aio-stress was doing O_DIRECT writes. I really don't have any real world example, I just cooked up a hypothetical scenario.

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

Thanks
Vivek

#9 lore raw
Quoted 3 lines · Vivek Goyal

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

They're both wrong. Ext4 completely misses support in fsync or sync to catch pending unwrittent extent conversions, and thus fails to obey the data integrity guarante. XFS is beeing rather stupid about the amount of synchronization it requires. The untested patch below should help with avoiding the synchronization if you're purely doing overwrites:

fs/xfs/linux-2.6/xfs_aops.c +15 -3
Index: xfs/fs/xfs/linux-2.6/xfs_aops.c
===================================================================
@@ -270,7 +270,7 @@ xfs_finish_ioend_sync(
  * (vs. incore size).
  */
 STATIC xfs_ioend_t *
-xfs_alloc_ioend(
+__xfs_alloc_ioend(
 	struct inode		*inode,
 	unsigned int		type)
 {
@@ -290,7 +290,6 @@ xfs_alloc_ioend(
 	ioend->io_inode = inode;
 	ioend->io_buffer_head = NULL;
 	ioend->io_buffer_tail = NULL;
-	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
 	ioend->io_offset = 0;
 	ioend->io_size = 0;
 	ioend->io_iocb = NULL;
@@ -300,6 +299,18 @@ xfs_alloc_ioend(
 	return ioend;
 }
 
+STATIC xfs_ioend_t *
+xfs_alloc_ioend(
+	struct inode		*inode,
+	unsigned int		type)
+{
+	struct xfs_ioend	*ioend;
+
+	ioend = __xfs_alloc_ioend(inode, type);
+	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
+	return ioend;
+}
+
 STATIC int
 xfs_map_blocks(
 	struct inode		*inode,
@@ -1318,6 +1329,7 @@ xfs_end_io_direct_write(
 	 */
 	iocb->private = NULL;
 
+	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
 	ioend->io_offset = offset;
 	ioend->io_size = size;
 	if (private && size > 0)
@@ -1354,7 +1366,7 @@ xfs_vm_direct_IO(
 	ssize_t			ret;
 
 	if (rw & WRITE) {
-		iocb->private = xfs_alloc_ioend(inode, IO_DIRECT);
+		iocb->private = __xfs_alloc_ioend(inode, IO_DIRECT);
 
 		ret = __blockdev_direct_IO(rw, iocb, inode, bdev, iov,
 					    offset, nr_segs,
#10 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 3 lines · Vivek Goyal

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

They're both wrong. Ext4 completely misses support in fsync or sync to catch pending unwrittent extent conversions, and thus fails to obey the data integrity guarante. XFS is beeing rather stupid about the amount of synchronization it requires. The untested patch below should help with avoiding the synchronization if you're purely doing overwrites:

Yes this patch helps. I have already laid out the file and doing overwrites.

I throttled aio-stress in one cgroup to 1 byte/sec and edited another file from other cgroup and did a "sync" and it completed.

Thanks
Vivek

Quoted 59 lines

Index: xfs/fs/xfs/linux-2.6/xfs_aops.c
===================================================================

fs/xfs/linux-2.6/xfs_aops.c +15 -3
@@ -270,7 +270,7 @@ xfs_finish_ioend_sync(
  * (vs. incore size).
  */
 STATIC xfs_ioend_t *
-xfs_alloc_ioend(
+__xfs_alloc_ioend(
 	struct inode		*inode,
 	unsigned int		type)
 {
@@ -290,7 +290,6 @@ xfs_alloc_ioend(
 	ioend->io_inode = inode;
 	ioend->io_buffer_head = NULL;
 	ioend->io_buffer_tail = NULL;
-	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
 	ioend->io_offset = 0;
 	ioend->io_size = 0;
 	ioend->io_iocb = NULL;
@@ -300,6 +299,18 @@ xfs_alloc_ioend(
 	return ioend;
 }
 
+STATIC xfs_ioend_t *
+xfs_alloc_ioend(
+	struct inode		*inode,
+	unsigned int		type)
+{
+	struct xfs_ioend	*ioend;
+
+	ioend = __xfs_alloc_ioend(inode, type);
+	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
+	return ioend;
+}
+
 STATIC int
 xfs_map_blocks(
 	struct inode		*inode,
@@ -1318,6 +1329,7 @@ xfs_end_io_direct_write(
 	 */
 	iocb->private = NULL;
 
+	atomic_inc(&XFS_I(ioend->io_inode)->i_iocount);
 	ioend->io_offset = offset;
 	ioend->io_size = size;
 	if (private && size > 0)
@@ -1354,7 +1366,7 @@ xfs_vm_direct_IO(
 	ssize_t			ret;
 
 	if (rw & WRITE) {
-		iocb->private = xfs_alloc_ioend(inode, IO_DIRECT);
+		iocb->private = __xfs_alloc_ioend(inode, IO_DIRECT);
 
 		ret = __blockdev_direct_IO(rw, iocb, inode, bdev, iov,
 					    offset, nr_segs,
#11 lore raw
Quoted 13 lines · Vivek Goyal
Quoted 11 lines · Christoph Hellwig
Quoted 3 lines · Vivek Goyal

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

They're both wrong. Ext4 completely misses support in fsync or sync to catch pending unwrittent extent conversions, and thus fails to obey the data integrity guarante. XFS is beeing rather stupid about the amount of synchronization it requires. The untested patch below should help with avoiding the synchronization if you're purely doing overwrites:

Quoted 5 lines

Yes this patch helps. I have already laid out the file and doing overwrites.

I throttled aio-stress in one cgroup to 1 byte/sec and edited another file from other cgroup and did a "sync" and it completed.

Even other test where I am running aio-stress in one window and edited a file in another window and typed "sync" worked. "sync" does not hang waiting for aio-stress to finish.

Thanks
Vivek

#12 lore raw
Quoted 9 lines · Vivek Goyal
Quoted 5 lines

Yes this patch helps. I have already laid out the file and doing overwrites.

I throttled aio-stress in one cgroup to 1 byte/sec and edited another file from other cgroup and did a "sync" and it completed.

Even other test where I am running aio-stress in one window and edited a file in another window and typed "sync" worked. "sync" does not hang waiting for aio-stress to finish.

I've been thinking about the patch a bit more, and I think it's simply incorrect. i_iocount is the only thing that actually tracks in-flight DIO/AIO requests, so we can't actually skip incrementing it as that means we can't wait for pending AIO in fsync/sync/inode reclaim or remount r/o.

We could simply declare AIO is off limits for sync and skip it there, which is easily doable, but we'd still need a special case version of sync for remount r/o as that absolutely needs to stop all pending I/O.

Of the other filesystem ext4 also has the counter, but only waits for it during inode teardown, and using a slightly different, but also effective scheme for fsync, but completely ignores sync and remount.

I couldn't find a similar scheme in other filesystem supporting AIO, but it might be hidden a bit better.

I suspect we could optimize things by using the dual count and list approach ext4 does - there is a counter for in-flight direct I/O, which we only check for inode teardown and remount, as those need to stop pending I/O, but sync and fsync can skip them as they only need to flush pending I/O. There is a list for the pending unwritten extent conversions that only gets appended to when the actual I/O is done, and the unwritten extent conversion is queued up.

I'll see if I can come up with a good scheme for that, preferably sitting directly in the direct I/O code, so that everyone gets it without additional work.

#13 lore raw
In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Vivek Goyal

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

They're both wrong. Ext4 completely misses support in fsync or sync to catch pending unwrittent extent conversions, and thus fails to obey the data integrity guarante.

I'm not sure about that.

ext4_sync_file() does ext4_flush_completed_IO() which does:

* When IO is completed, the work to convert unwritten extents to
* written is queued on workqueue but may not get immediately
* scheduled. When fsync is called, we need to ensure the
* conversion is complete before fsync returns.
* The inode keeps track of a list of pending/completed IO that
* might needs to do the conversion. This function walks through
* the list and convert the related unwritten extents for completed IO
* to written.

Granted, I get easily lost in ext4's codepaths here, which is actually why I suggested Vivek pose these questions to the list ;)

-Eric

#14 lore raw
Quoted 10 lines · Eric Sandeen
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Vivek Goyal

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

They're both wrong. Ext4 completely misses support in fsync or sync to catch pending unwrittent extent conversions, and thus fails to obey the data integrity guarante.

Quoted 6 lines

I'm not sure about that.

ext4_sync_file() does ext4_flush_completed_IO() which does:

Granted, I get easily lost in ext4's codepaths here, which is actually why I suggested Vivek pose these questions to the list ;)

You're right it gets fsync right, but the sync still seems to be missing, which does not just include sync, but also the syncfs system call and unmount.

#15 lore raw
In reply to Vivek Goyal
Quoted 8 lines · Vivek Goyal

In this case I had done "sync" while aio-stress was doing O_DIRECT writes. I really don't have any real world example, I just cooked up a hypothetical scenario.

Just wondering why ext4 and XFS behavior are different and which is a more appropriate behavior. ext4 does not seem to be waiting for all pending AIO/DIO to finish while XFS does.

I think this is something we can chalk up to "different implementations do different things". I'm not convinced either behviour is wrong per se. Granted, the recent work to make sync and fsync not livelock in the face of continuing writes means that I'm more happy with ext4's behaviour, but I don't think that means xfs's behavior is wrong.

One of the things that I have thought about is sysctl which makes sync a no-op unless you are root. The reason for that is that many system administrators sometimes have a habit of typing sync, and on a heavily loaded production server, this can really cause performance to go to hell for up to tens of minutes. So it might make sense to not allow non-root users from trashing overall system performance by running sync....

- Ted

#16 lore raw
In reply to Vivek Goyal
Quoted 15 lines · Vivek Goyal
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Vivek Goyal

Hi,

If I throttle a DIO/AIO WRITE bio at block device in a cgroup, will it lead to any kind of serialization of ext4 file system. IOW, is there any filesystem operation which will wait for that DIO/AIO WRITE to finish before other filesystem can make progress (fsync, journalling etc?)

Quoted 5 lines

Truncate?

(XFS explicitly serialises truncate against in flight DIO, regardless of whether ext4 does.)

Quoted 6 lines

Dave,

Does this serialization happens against that particular inode on which truncate has been called? If yes, then I think I will still be fine as in common use case I am not expecting much sharing of inodes across cgroups.

Same inode serialisation only.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]