Skip to contentSkip to conversation

gfs2 write bandwidth regression on 6.4-rc3 compareto 5.15.y

DISCUSSION Wang Yugui 3 participants 6 messages May 23, 2023 – Jul 11, 2023 VFS

lore
Share & tools
lore

Wang Yugui

author
#0 lore raw

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y becasue some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
	-name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
	-name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
	-name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
	-name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.

5.15.y result

	fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
	fio  WRITE: bw=2599MiB/s (2725MB/s)

Best Regards
Wang Yugui (wangyugui at e16-tech.com)
2023/05/23

author
#1 lore raw

Hi,

Quoted 29 lines

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y becasue some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
	-name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
	-name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
	-name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
	-name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.
Quoted 4 lines

5.15.y result

	fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
	fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

5.17.0 WRITE: bw=4988MiB/s (5231MB/s)
5.18.0 WRITE: bw=5165MiB/s (5416MB/s)
5.19.0 WRITE: bw=5511MiB/s (5779MB/s)
6.0.5 WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
6.1.30 WRITE: bw=2579MiB/s (2705MB/s)

so this regression happen in some code introduced in 6.0, and maybe some minor regression in 6.1 too?

Best Regards
Wang Yugui (wangyugui at e16-tech.com)
2023/05/28

#2 lore raw

Hi Wang Yugui,

Quoted 38 lines · On Sun, May 28, 2023 at 5:53?PM Wang Yugui <[email protected]> wrote:

Hi,

Quoted 29 lines

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
      -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
      -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
      -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
      -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.
Quoted 4 lines

5.15.y result

      fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
      fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

Quoted 8 lines
5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
6.1.30  WRITE: bw=2579MiB/s (2705MB/s)

so this regression happen in some code introduced in 6.0, and maybe some minor regression in 6.1 too?

thanks for this bug report. Bob has noticed a similar looking performance regression recently, and it turned out that commit e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now") inadvertently caused buffered writes to fall back to writing single pages instead of multiple pages at once. That patch was added in v5.18, so it doesn't perfectly align with the regression history you're reporting, but maybe there's something else going on that we're not aware of.

In any case, the regression introduced by commit e1fa9ea85ce8 should be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate should_fault_in_pages() call"), which ended up in v6.5-rc1.

Could you please check where we end up with that fix?

Thank you very much,
Andreas

author
#3 lore raw

Hi,

Quoted 59 lines

Hi Wang Yugui,

Quoted 38 lines · On Sun, May 28, 2023 at 5:53?PM Wang Yugui <[email protected]> wrote:

Hi,

Quoted 29 lines

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
      -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
      -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
      -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
      -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.
Quoted 4 lines

5.15.y result

      fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
      fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

Quoted 8 lines
5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
6.1.30  WRITE: bw=2579MiB/s (2705MB/s)

so this regression happen in some code introduced in 6.0, and maybe some minor regression in 6.1 too?

thanks for this bug report. Bob has noticed a similar looking performance regression recently, and it turned out that commit e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now") inadvertently caused buffered writes to fall back to writing single pages instead of multiple pages at once. That patch was added in v5.18, so it doesn't perfectly align with the regression history you're reporting, but maybe there's something else going on that we're not aware of.

Quoted 5 lines

In any case, the regression introduced by commit e1fa9ea85ce8 should be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate should_fault_in_pages() call"), which ended up in v6.5-rc1.

Could you please check where we end up with that fix?

I applied c8ed1b359312 on 6.1.36.

# the build/test of 6.5-rc1 is yet not ready.

fio performance result:
WRITE: bw=2683MiB/s (2813MB/s)

but the performance of fio 'Laying out IO file' is improved.
Jobs: 4 (f=4): [F(4)][100.0%][w=5168MiB/s][w=5168 IOPS][eta 00m:00s]

so there seems 2 problems, one is fixed by c8ed1b359312. but another is still left.

Best Regards
Wang Yugui (wangyugui at e16-tech.com)
2023/07/11

author
#4 lore raw

Hi,

Quoted 70 lines

Hi,

Quoted 59 lines

Hi Wang Yugui,

Quoted 38 lines · On Sun, May 28, 2023 at 5:53?PM Wang Yugui <[email protected]> wrote:

Hi,

Quoted 29 lines

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
      -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
      -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
      -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
      -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.
Quoted 4 lines

5.15.y result

      fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
      fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

Quoted 8 lines
5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
6.1.30  WRITE: bw=2579MiB/s (2705MB/s)

so this regression happen in some code introduced in 6.0, and maybe some minor regression in 6.1 too?

thanks for this bug report. Bob has noticed a similar looking performance regression recently, and it turned out that commit e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now") inadvertently caused buffered writes to fall back to writing single pages instead of multiple pages at once. That patch was added in v5.18, so it doesn't perfectly align with the regression history you're reporting, but maybe there's something else going on that we're not aware of.

Quoted 5 lines

In any case, the regression introduced by commit e1fa9ea85ce8 should be fixed by commit c8ed1b359312 ("gfs2: Fix duplicate should_fault_in_pages() call"), which ended up in v6.5-rc1.

Could you please check where we end up with that fix?

I applied c8ed1b359312 on 6.1.36.

# the build/test of 6.5-rc1 is yet not ready.
Quoted 8 lines

fio performance result:
WRITE: bw=2683MiB/s (2813MB/s)

but the performance of fio 'Laying out IO file' is improved.
Jobs: 4 (f=4): [F(4)][100.0%][w=5168MiB/s][w=5168 IOPS][eta 00m:00s]

so there seems 2 problems, one is fixed by c8ed1b359312. but another is still left.

We build/test on 6.5-rc1 too.
fio WRITE: bw=2643MiB/s (2771MB/s)

the performance of fio 'Laying out IO file'.
Jobs: 4 (f=4): [F(4)][100.0%][w=4884MiB/s][w=4884 IOPS][eta 00m:00s]

so the performance result on 6.5-rc1 is the same level as 6.1.36 with c8ed1b359312.

Best Regards
Wang Yugui (wangyugui at e16-tech.com)
2023/07/11

#5 lore raw
In reply to Andreas Gruenbacher
Quoted 50 lines · Andreas Gruenbacher

Hi Wang Yugui,

Quoted 38 lines · On Sun, May 28, 2023 at 5:53?PM Wang Yugui <[email protected]> wrote:

Hi,

Quoted 29 lines

Hi,

gfs2 write bandwidth regression on 6.4-rc3 compare to 5.15.y.

we added linux-xfs@ and linux-fsdevel@ because some related problem[1] and related patches[2].

we compared 6.4-rc3(rather than 6.1.y) to 5.15.y because some related patches[2] work only for 6.4 now.

[1] https://lore.kernel.org/linux-xfs/20230508172406.1CF3.409509F4 at e16-tech.com/
[2] https://lore.kernel.org/linux-xfs/20230520163603.1794256-1-willy at infradead.org/

test case:
1) PCIe3 SSD *4 with LVM
2) gfs2 lock_nolock
gfs2 attr(T) GFS2_AF_ORLOV

   # chattr +T /mnt/test
3) fio
fio --name=global --rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
-ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=1 \
      -name write-bandwidth-1 -filename=/mnt/test/sub1/1.txt \
      -name write-bandwidth-2 -filename=/mnt/test/sub2/1.txt \
      -name write-bandwidth-3 -filename=/mnt/test/sub3/1.txt \
      -name write-bandwidth-4 -filename=/mnt/test/sub4/1.txt
4) patches[2] are applied to 6.4-rc3.
Quoted 4 lines

5.15.y result

      fio WRITE: bw=5139MiB/s (5389MB/s),
6.4-rc3 result
      fio  WRITE: bw=2599MiB/s (2725MB/s)

more test result:

Quoted 8 lines
5.17.0  WRITE: bw=4988MiB/s (5231MB/s)
5.18.0  WRITE: bw=5165MiB/s (5416MB/s)
5.19.0  WRITE: bw=5511MiB/s (5779MB/s)
6.0.5   WRITE: bw=3055MiB/s (3203MB/s), WRITE: bw=3225MiB/s (3382MB/s)
6.1.30  WRITE: bw=2579MiB/s (2705MB/s)

so this regression happen in some code introduced in 6.0, and maybe some minor regression in 6.1 too?

Quoted 8 lines

thanks for this bug report. Bob has noticed a similar looking performance regression recently, and it turned out that commit e1fa9ea85ce8 ("gfs2: Stop using glock holder auto-demotion for now") inadvertently caused buffered writes to fall back to writing single pages instead of multiple pages at once. That patch was added in v5.18, so it doesn't perfectly align with the regression history you're reporting, but maybe there's something else going on that we're not aware of.

Dave gave a good explanation of the problem here:

https://lore.kernel.org/linux-xfs/ZKybxCxzmuI1TFYn at dread.disaster.area/

It's a pagecache locking contention problem rather than an individual filesystem problem.

... are you interested in supporting large folios in gfs2? ;-)