Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

Hi all,

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

Because the slowdown has been here for few kernel releases I suppose nobody complained and the test is somehow artificial (nobody uses this in a real world). But still it'd be good to double check the problem. I can bisect a particular commit.

Because 2 filesystems affected, could be "Improve asynchronous iomap DIO performance" [4] block layer change somehow related?

Quoted 16 lines

Hi all,

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

Because the slowdown has been here for few kernel releases I suppose nobody complained and the test is somehow artificial (nobody uses this in a real world). But still it'd be good to double check the problem. I can bisect a particular commit.

Quoted 2 lines

Because 2 filesystems affected, could be "Improve asynchronous iomap DIO performance" [4] block layer change somehow related?

No, because that got disabled before release for unrelated reasons. Why don't you just bisect it, since you have a simple test case?

Hi Jens, all,

Quoted 18 lines · Petr Vorel

Quoted 16 lines

Hi all,

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

Because the slowdown has been here for few kernel releases I suppose nobody complained and the test is somehow artificial (nobody uses this in a real world). But still it'd be good to double check the problem. I can bisect a particular commit.

Quoted 5 lines

Quoted 2 lines

Because 2 filesystems affected, could be "Improve asynchronous iomap DIO performance" [4] block layer change somehow related?

No, because that got disabled before release for unrelated reasons. Why don't you just bisect it, since you have a simple test case?

Jens, thanks for info. Sure, I'll bisect next week and report.

The reason I reported before bisecting is because it wouldn't be the first time the test was "artificial" and therefore reported problem was not fixed. If it's a real problem I would expect it would be also caught by other people or even by fstests.

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

Hi!

Quoted 5 lines · Petr Vorel

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

So the performance of this test is irrelevant because combining buffered reads with direct IO writes was always in "better don't do it" territory. Definitely not if you care about perfomance.

Quoted 4 lines

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

But still it's kind of curious what caused the 9x slow down. So I'd be curious to know the result of the bisection :). Thanks for report!

Honza

Hi!

Quoted 5 lines · Petr Vorel

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

So the performance of this test is irrelevant because combining buffered reads with direct IO writes was always in "better don't do it" territory. Definitely not if you care about perfomance.

Quoted 4 lines

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

But still it's kind of curious what caused the 9x slow down. So I'd be curious to know the result of the bisection :). Thanks for report!

Honza

Hi all,

[ Cc Peter and Mike ]

Quoted 17 lines

Hi!

Quoted 5 lines · Petr Vorel

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while 16 other processes reads the buffer and check only zero is there.

So the performance of this test is irrelevant because combining buffered reads with direct IO writes was always in "better don't do it" territory. Definitely not if you care about perfomance.

Quoted 4 lines

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected. (Non default parameter creates much smaller file, thus the change is not that obvious).

Quoted 2 lines

But still it's kind of curious what caused the 9x slow down. So I'd be curious to know the result of the bisection :). Thanks for report!

It looks to be the slowdown was introduced by commit 63304558ba5d ("sched/eevdf: Curb wakeup-preemption") [1] from v6.6-rc1.