High write iops

Hello,

using Ubuntu server 18.04.2 btrfs on /

10 absolutely identical servers (Dell M630), with the same configuration, updated 2/10 and encountered this problem on both.
After upgrading kernel from version 4.15.0-36-generic # 39-Ubuntu SMP observing at the same software configuration an increased load on the disk (write) - MySQL previously loaded the disk at 20-50 mB/s, after the update it started loading at 150-250 mB/s.

MySQL configuration and package version were not changed.
If I boot into the old kernel version (4.15.0-36), the problem disappears. Also tested kernels:
4.15.0-52 - problem persists
4.18.0-25 (hwe) - problem persists, load became more 200-250 mB/s
5.2-rc7 (vanilla) - problem persists
Changing /var/lib/mysql partition to ext4 fixes the problem.

With the help of Kai-Heng Feng in bugs.launchpad.net founded that this commit causes the regression:
commit 31d11b83b96faaee4bb514d375a09489117c3e8d
Author: Filipe Manana < [email protected] >
Date: Wed May 9 16:01:46 2018 +0100
Btrfs: fix duplicate extents after fsync of file with prealloc extents

All the details you can find in the bug/1835185 at bugs.launchpad.net

Igor Naumov

Have you set nodatacow on MySQL data filess? Also have you seen actual increase of latency of your mysql queries because it's possible that this commit improved what fsync does and allowed your filesystem to write more data per unit of time? Have you seen drop in IOPS?