very poor performance / a lot of writes to disk with space_cache (but not with space_cache=v2)
I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.
The slave receives around 0.5-1 MB/s of data from the master over the network, which is then saved to MySQL's relay log and executed. In ideal conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s of data written to disk.
MySQL directory and files in it are chattr +C (since the directory was created, so all files are really +C); there are no snapshots.
Now, an interesting thing.
When the filesystem is mounted with these options in fstab:
defaults,noatime,discard
We can see a *constant* write of 25-100 MB/s to each disk. The system is generally unresponsive and it sometimes takes long seconds for a simple command executed in bash to return.
However, as soon as we remount the filesystem with space_cache=v2 - writes drop to just around 3-10 MB/s to each disk. If we remount to space_cache - lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.
That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Tomasz Chmielewski
https://lxadm.com
Quoted 2 lines · Tomasz Chmielewski
I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.
This sounds a little concerning.
Not about the the usage percentage itself. but the size and how many free space cache could be updated for each transaction.
Detail will follow below.
Quoted 8 lines
The slave receives around 0.5-1 MB/s of data from the master over the network, which is then saved to MySQL's relay log and executed. In ideal conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s of data written to disk.
MySQL directory and files in it are chattr +C (since the directory was created, so all files are really +C); there are no snapshots.
Not familiar with space cache nor MySQL workload, but at least we don't need to bother extra data CoW.
Quoted 4 lines
Now, an interesting thing.
Quoted 8 lines
When the filesystem is mounted with these options in fstab:
defaults,noatime,discard
We can see a *constant* write of 25-100 MB/s to each disk. The system is generally unresponsive and it sometimes takes long seconds for a simple command executed in bash to return.
The main concern here is how many metadata block groups are involved in one transaction.
From my observation, although free space cache files (v1 space cache) are marked NODATACOW, they in fact get updated in a COW behavior.
This means if there are say 100 metadata block groups get updated, then we need to write around 12M data just for space cache.
On the other than, if we fix v1 space cache to really do NODATACOW, then it should hugely reduce the IO for free space cache
Quoted 6 lines
However, as soon as we remount the filesystem with space_cache=v2 - writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.
Have you tried nospace_cache? I think it should behavior a little worse than v2 space cache but much better than the *broken* v1 space cache.
And for v2 space cache, it's already based on btrfs btree, which get CoWed like all other btrfs btrees, thus no need to update the whole space cache for each metadata block group. (Although in theory, the overhead should still be larger than *working* v1 cache)
Thanks,
Qu
Quoted 8 lines
That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Tomasz Chmielewski
https://lxadm.com
Quoted 10 lines · Tomasz Chmielewski
I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.
The slave receives around 0.5-1 MB/s of data from the master over the network, which is then saved to MySQL's relay log and executed. In ideal conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s of data written to disk.
MySQL directory and files in it are chattr +C (since the directory was created, so all files are really +C); there are no snapshots.
Not related to the issue you are reporting, but I thought it's worth mentioning, (since not many do), that using chattr +C on a BTRFS Raid 1 is a dangerous thing. without COW, the 2 copies are never synchronized, even if a scrub is executed. So any kind of unclean shutdown that interrupts writes (not to mention the extreme of a temporarily disconnected drive.) will result in files that are inconsistent. (ie, depending on which disk happens to read at the time, the data will be different on each read.)
begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:[email protected]
tel;work:226-256-1545
version:2.1
end:vcard
Hi,
Quoted 14 lines · Tomasz Chmielewski
I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.
The slave receives around 0.5-1 MB/s of data from the master over the network, which is then saved to MySQL's relay log and executed. In ideal conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s of data written to disk.
MySQL directory and files in it are chattr +C (since the directory was created, so all files are really +C); there are no snapshots.
Now, an interesting thing.
Quoted 7 lines
When the filesystem is mounted with these options in fstab:
defaults,noatime,discard
We can see a *constant* write of 25-100 MB/s to each disk. The system is generally unresponsive and it sometimes takes long seconds for a simple command executed in bash to return.
Did you already test the difference with/without 'discard'? Also, I think that depending on the tooling that you use to view disk IO, discards will also show up as disk write statistics.
Quoted 7 lines
However, as soon as we remount the filesystem with space_cache=v2 - writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Yes, that does not surprise me.
https://events.static.linuxfound.org/sites/events/files/slides/vault2016_0.pdf
Free space cache v1 is the default because of issues with btrfs-progs, not because it's unwise to use the kernel code. I can totally recommend using it. The linked presentation above gives some good background information.
Another thing that's interesting is finding out what kind of things btrfs is writing if it's writing that much MB/s to disk. Doing this is not very trivial.
I've been spending quite some time researching these kind of issues.
Here's what I found out:
https://www.spinics.net/lists/linux-btrfs/msg70624.html (oh wow, that's almost a year ago already)
There are a bunch of tracepoints in the kernel code that could help debugging all of this more, but I've not yet gotten around to writing something to conveniently to use them to live show what's happening.
I'm still using the "Thanks to a bug, solved in [2]" in the above mailing list post way of combining extent allocators in btrfs now to keep things workable on the larger filesystem.
Signature · 1 line
Hans van Kranenburg
Hans van Kranenburg - 19.09.18, 19:58:
Quoted 11 lines
However, as soon as we remount the filesystem with space_cache=v2 -
Quoted 6 lines
writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Yes, that does not surprise me.
Quoted 7 lines
https://events.static.linuxfound.org/sites/events/files/slides/vault20
16_0.pdfFree space cache v1 is the default because of issues with btrfs-progs, not because it's unwise to use the kernel code. I can totally recommend using it. The linked presentation above gives some good background information.
What issues in btrfs-progs are that?
I am wondering whether to switch to freespace tree v2. Would it provide benefit for a regular / and /home filesystems as dual SSD BTRFS RAID-1 on a laptop?
Thanks,
Signature · 1 line
Martin
Quoted 15 lines · Martin Steigerwald
Hans van Kranenburg - 19.09.18, 19:58:
Quoted 14 lines
However, as soon as we remount the filesystem with space_cache=v2 -
Quoted 6 lines
writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Yes, that does not surprise me.
https://events.static.linuxfound.org/sites/events/files/slides/vault20
16_0.pdf
Quoted 6 lines
Quoted 4 lines
Free space cache v1 is the default because of issues with btrfs-progs, not because it's unwise to use the kernel code. I can totally recommend using it. The linked presentation above gives some good background information.
What issues in btrfs-progs are that?
Missing code to make offline changes to a filesystem that has a free space tree. So when using btrfstune / repair / whatever you first need to remove the whole free space tree with a command, and then add it back on the next mount.
For me personally that's not a problem (I don't have to make offline changes), but I understand that having that situation out of the box for every new user would be a bit awkward.
Quoted 3 lines
I am wondering whether to switch to freespace tree v2. Would it provide benefit for a regular / and /home filesystems as dual SSD BTRFS RAID-1 on a laptop?
As shown in the linked presentation, it provides benefit on a largeish filesystem and if your writes are touching a lot of different block groups (since v1 writes out the full space cache for all of them on every transaction commit). I'd say, it provides benefit as soon as you encounter filesystem delays because of it, and as soon as you see using it eases the pain a lot. So, yes, that's your case.
Signature · 1 line
Hans van Kranenburg
Quoted 21 lines · Hans van Kranenburg
Quoted 20 lines · Martin Steigerwald
Hans van Kranenburg - 19.09.18, 19:58:
Quoted 18 lines
However, as soon as we remount the filesystem with space_cache=v2 -
Quoted 6 lines
writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Yes, that does not surprise me.
https://events.static.linuxfound.org/sites/events/files/slides/vault20
16_0.pdfFree space cache v1 is the default because of issues with btrfs-progs, not because it's unwise to use the kernel code. I can totally recommend using it. The linked presentation above gives some good background information.
Quoted 6 lines
Quoted 1 line
What issues in btrfs-progs are that?
Missing code to make offline changes to a filesystem that has a free space tree. So when using btrfstune / repair / whatever you first need to remove the whole free space tree with a command, and then add it back on the next mount.
And as a matter of fact this code has already been published on the mailing list for review, and even some parts got merged so we are in good shape to get it into progs and eventually switch the kernel to default to v2 cache.
Quoted 16 lines
For me personally that's not a problem (I don't have to make offline changes), but I understand that having that situation out of the box for every new user would be a bit awkward.
Quoted 3 lines
I am wondering whether to switch to freespace tree v2. Would it provide benefit for a regular / and /home filesystems as dual SSD BTRFS RAID-1 on a laptop?
As shown in the linked presentation, it provides benefit on a largeish filesystem and if your writes are touching a lot of different block groups (since v1 writes out the full space cache for all of them on every transaction commit). I'd say, it provides benefit as soon as you encounter filesystem delays because of it, and as soon as you see using it eases the pain a lot. So, yes, that's your case.
Quoted 32 lines · Hans van Kranenburg
Quoted 15 lines · Martin Steigerwald
Hans van Kranenburg - 19.09.18, 19:58:
Quoted 14 lines
However, as soon as we remount the filesystem with space_cache=v2 -
Quoted 6 lines
writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
Yes, that does not surprise me.
https://events.static.linuxfound.org/sites/events/files/slides/vault20
16_0.pdfQuoted 6 lines
Quoted 4 lines
Free space cache v1 is the default because of issues with btrfs-progs, not because it's unwise to use the kernel code. I can totally recommend using it. The linked presentation above gives some good background information.
What issues in btrfs-progs are that?
Missing code to make offline changes to a filesystem that has a free space tree. So when using btrfstune / repair / whatever you first need to remove the whole free space tree with a command, and then add it back on the next mount.
For me personally that's not a problem (I don't have to make offline changes), but I understand that having that situation out of the box for every new user would be a bit awkward.
Quoted 8 lines
Quoted 3 lines
I am wondering whether to switch to freespace tree v2. Would it provide benefit for a regular / and /home filesystems as dual SSD BTRFS RAID-1 on a laptop?
As shown in the linked presentation, it provides benefit on a largeish filesystem and if your writes are touching a lot of different block groups (since v1 writes out the full space cache for all of them on every transaction commit).
In fact that's the problem.
From free space cache inode flags, it's
NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC.
But the fact is, if it's modified, the whole file just get CoWed.
If we could change it to follow the inode flags, we could reduce overhead even smaller than v2 one.
(v1 needs at least (1 + n) * sectorsize(4K) one for the header which contains the csum, while v2 needs metadata CoW which is at least nodesize (default to 16K)).
Thanks,
Qu
Quoted 4 lines
I'd say, it provides benefit as soon as you
encounter filesystem delays because of it, and as soon as you see using it eases the pain a lot. So, yes, that's your case.
Tomasz Chmielewski posted on Wed, 19 Sep 2018 10:43:18 +0200 as excerpted:
Quoted 24 lines
I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.
The slave receives around 0.5-1 MB/s of data from the master over the network, which is then saved to MySQL's relay log and executed. In ideal conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s of data written to disk.
MySQL directory and files in it are chattr +C (since the directory was created, so all files are really +C); there are no snapshots.
Now, an interesting thing.
When the filesystem is mounted with these options in fstab:
defaults,noatime,discard
We can see a *constant* write of 25-100 MB/s to each disk. The system is generally unresponsive and it sometimes takes long seconds for a simple command executed in bash to return.
Quoted 8 lines
However, as soon as we remount the filesystem with space_cache=v2 - writes drop to just around 3-10 MB/s to each disk. If we remount to
space_cache- lots of writes, system unresponsive. Again remount to space_cache=v2 - low writes, system responsive.That's a huuge, 10x overhead! Is it expected? Especially that space_cache=v1 is still the default mount option?
The other replies are good but I've not seen this pointed out yet...
Perhaps you are accounting for this already, but you don't /say/ you are, while you do mention repeatedly toggling the space-cache options, which would trigger it so you /need/ to account for it...
I'm not sure about space_cache=v2 (it's probably more efficient with it even if it does have to do it), but I'm quite sure that space_cache=v1 takes some time after initial mount with it to scan the filesystem and actually create the map of available free space that is the space_cache.
Now you said ssds, which should be reasonably fast, but you also say 3- device btrfs raid1, with each device ~2TB, and the filesystem ~40% full, which should be ~2 TB of data, which is likely somewhat fragmented so it's likely rather more than 2 TB of data chunks to scan for free space, and that's going to take /some/ time even on SSDs!
So if you're toggling settings like that in your tests, be sure to let the filesystem rebuild its cache that you just toggled and give it time to complete that and quiesce, before you start trying to measure write amplification.
Otherwise it's not write-amplification you're measuring, but the churn from the filesystem still trying to reset its cache after you toggled it!
Also, while 4.17 is well after the ssd mount option (usually auto- detected, check /proc/mounts, mount output, or dmesg, to see if the ssd mount option is being added) fixes that went in in 4.14, if the filesystem has been in use for several kernel cycles and in particular before 4.14, with the ssd mount option active, and you've not rebalanced since then, you may well still have serious space fragmentation from that, which could increase the amount of data in the space_cache map rather drastically, thus increasing the time it takes to update the space_cache, particularly v1, after toggling it on.
A balance can help correct that, but it might well be easier and should result in a better layout to simply blow the filesystem away with a mkfs.btrfs and start over.
Meanwhile, as Remi already mentioned, you might want to reconsider nocow on btrfs raid1, since nocow defeats checksumming and thus scrub, which verifies checksums, simply skips it, and if the two copies get out of sync for some reason...
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman