writeback: plug writeback at a high level
PATCH 1/1 writeback: plug writeback at a high level
From: Dave Chinner <[email protected]>
Doing writeback on lots of little files causes terrible IOPS storms because of the per-mapping writeback plugging we do. This essentially causes imeediate dispatch of IO for each mapping, regardless of the context in which writeback is occurring.
IOWs, running a concurrent write-lots-of-small 4k files using fsmark on XFS results in a huge number of IOPS being issued for data writes. Metadata writes are sorted and plugged at a high level by XFS, so aggregate nicely into large IOs. However, data writeback IOs are dispatched in individual 4k IOs, even when the blocks of two consecutively written files are adjacent.
Test VM: 8p, 8GB RAM, 4xSSD in RAID0, 100TB sparse XFS filesystem, metadata CRCs enabled.
Kernel: 3.10-rc5 + xfsdev + my 3.11 xfs queue (~70 patches)
Test:
$ ./fs_mark -D 10000 -S0 -n 10000 -s 4096 -L 120 -d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/2 -d /mnt/scratch/3 -d /mnt/scratch/4 -d /mnt/scratch/5 -d /mnt/scratch/6 -d /mnt/scratch/7
Result:
wall sys create rate Physical write IO
time CPU (avg files/s) IOPS Bandwidth
----- ----- ------------ ------ ---------
unpatched 6m56s 15m47s 24,000+/-500 26,000 130MB/s
patched 5m06s 13m28s 32,800+/-600 1,500 180MB/s
improvement -26.44% -14.68% +36.67% -94.23% +38.46%
If I use zero length files, this workload at about 500 IOPS, so plugging drops the data IOs from roughly 25,500/s to 1000/s. 3 lines of code, 35% better throughput for 15% less CPU.
The benefits of plugging at this layer are likely to be higher for spinning media as the IO patterns for this workload are going make a much bigger difference on high IO latency devices.....
--- fs/fs-writeback.c | 3 +++ 1 file changed, 3 insertions(+)
fs/fs-writeback.c +3 -0
@@ -585,7 +585,9 @@ static long writeback_sb_inodes(struct super_block *sb,unsigned long start_time = jiffies;long write_chunk;long wrote =0 ;/* count both pages and inodes */ +struct blk_plug plug; +blk_start_plug (&plug);while (!list_empty (&wb->b_io)) {struct inode *inode =wb_inode (wb->b_io.prev);
@@ -682,6 +684,7 @@ static long writeback_sb_inodes(struct super_block *sb,break ; } } +blk_finish_plug (&plug);return wrote; }
Signature · 1 line
1.7.10.4
Quoting Dave Chinner (2013-06-14 22:50:50)
Quoted 14 lines
From: Dave Chinner <[email protected]>
Doing writeback on lots of little files causes terrible IOPS storms because of the per-mapping writeback plugging we do. This essentially causes imeediate dispatch of IO for each mapping, regardless of the context in which writeback is occurring.
IOWs, running a concurrent write-lots-of-small 4k files using fsmark on XFS results in a huge number of IOPS being issued for data writes. Metadata writes are sorted and plugged at a high level by XFS, so aggregate nicely into large IOs. However, data writeback IOs are dispatched in individual 4k IOs, even when the blocks of two consecutively written files are adjacent.
Quoted 4 lines
Test VM: 8p, 8GB RAM, 4xSSD in RAID0, 100TB sparse XFS filesystem, metadata CRCs enabled.
Kernel: 3.10-rc5 + xfsdev + my 3.11 xfs queue (~70 patches)
I'm a little worried about this one, just because of the impact on ssds from plugging in the aio code:
https://lkml.org/lkml/2011/12/13/326
How exactly was your FS created? I'll try it here.
-chris
Quoted 15 lines · Chris Mason
Quoting Dave Chinner (2013-06-14 22:50:50)
Quoted 14 lines
From: Dave Chinner <[email protected]>
Doing writeback on lots of little files causes terrible IOPS storms because of the per-mapping writeback plugging we do. This essentially causes imeediate dispatch of IO for each mapping, regardless of the context in which writeback is occurring.
IOWs, running a concurrent write-lots-of-small 4k files using fsmark on XFS results in a huge number of IOPS being issued for data writes. Metadata writes are sorted and plugged at a high level by XFS, so aggregate nicely into large IOs. However, data writeback IOs are dispatched in individual 4k IOs, even when the blocks of two consecutively written files are adjacent.
Quoted 7 lines
Quoted 4 lines
Test VM: 8p, 8GB RAM, 4xSSD in RAID0, 100TB sparse XFS filesystem, metadata CRCs enabled.
Kernel: 3.10-rc5 + xfsdev + my 3.11 xfs queue (~70 patches)
I'm a little worried about this one, just because of the impact on ssds from plugging in the aio code:
I'm testing on SSDs, but the impact has nothing to do with the underlying storage.
Quoted 1 line
Oh, that's completely different situation - it's application submitted IO wherhe Io latency is a deterimining factor in performance.
This is for background writeback, where IO latency is not a primary performance issue - maximum throughput is what we are trying to acheive here. For writeback, well formed IO has a much greater impact on throughput than low latency IO submission, even for SSD based storage.
Quoted 1 line
How exactly was your FS created? I'll try it here.
The host has an XFS filesystem on a md RAID0 of 4x40Gb slices off larger SSDs:
$ cat /proc/mdstat
Personalities : [raid0]
md2 : active raid0 sdb1[0] sde1[3] sdd1[2] sdc1[1]
167772032 blocks super 1.2 32k chunksbuilt with mkfs.xfs defaults. A sparse 100TB file is created and then fed to KVM with cache=none,virtio.
The guest formats the 100TB device using default mkfs.xfs parameters and uses default mount options, so it's a 100TB filesytsem with about 150GB of real storage in it...
The underlying hardware controller that the SSDs are attached to is limited to roughly 27,000 random 4k write IOPS, and that's the IO pattern that writeback is splattering at the device.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoting Dave Chinner (2013-06-17 21:58:06)
Quoted 33 lines · Chris Mason
Quoted 15 lines
Quoting Dave Chinner (2013-06-14 22:50:50)
Quoted 14 lines
From: Dave Chinner <[email protected]>
Doing writeback on lots of little files causes terrible IOPS storms because of the per-mapping writeback plugging we do. This essentially causes imeediate dispatch of IO for each mapping, regardless of the context in which writeback is occurring.
IOWs, running a concurrent write-lots-of-small 4k files using fsmark on XFS results in a huge number of IOPS being issued for data writes. Metadata writes are sorted and plugged at a high level by XFS, so aggregate nicely into large IOs. However, data writeback IOs are dispatched in individual 4k IOs, even when the blocks of two consecutively written files are adjacent.
Quoted 7 lines
Quoted 4 lines
Test VM: 8p, 8GB RAM, 4xSSD in RAID0, 100TB sparse XFS filesystem, metadata CRCs enabled.
Kernel: 3.10-rc5 + xfsdev + my 3.11 xfs queue (~70 patches)
I'm a little worried about this one, just because of the impact on ssds from plugging in the aio code:
I'm testing on SSDs, but the impact has nothing to do with the underlying storage.
Quoted 1 line
Oh, that's completely different situation - it's application submitted IO wherhe Io latency is a deterimining factor in performance.
Quoted 5 lines
This is for background writeback, where IO latency is not a primary performance issue - maximum throughput is what we are trying to acheive here. For writeback, well formed IO has a much greater impact on throughput than low latency IO submission, even for SSD based storage.
Very true, but at the same time we do wait for background writeback sometimes. It's worth a quick test...
Quoted 14 lines
Quoted 2 lines
How exactly was your FS created? I'll try it here.
The host has an XFS filesystem on a md RAID0 of 4x40Gb slices off larger SSDs:
$ cat /proc/mdstat Personalities : [raid0] md2 : active raid0 sdb1[0] sde1[3] sdd1[2] sdc1[1] 167772032 blocks super 1.2 32k chunksbuilt with mkfs.xfs defaults. A sparse 100TB file is created and then fed to KVM with cache=none,virtio.
Quoted 7 lines
The guest formats the 100TB device using default mkfs.xfs parameters and uses default mount options, so it's a 100TB filesytsem with about 150GB of real storage in it...
The underlying hardware controller that the SSDs are attached to is limited to roughly 27,000 random 4k write IOPS, and that's the IO pattern that writeback is splattering at the device.
Ok, I'll try something less exotic here ;)
-chris