btrfs send problems
Hi list - I'm having problems with btrfs send in general, and incremental send in particular.
1. Performance: in kernel 3.11, btrfs send would send data at 500+MB/sec from a Samsung 840 series solid state drive. In kernel 3.12 and up, btrfs send will only send 30-ish MB/sec from the same drive - though if you interrupt a btrfs send in progress, it will "catch up" to where it was at 500+ MB/sec. This is pretty weird and frustrating. Even weirder and more frustrating, even at 30-ish MB/sec, a btrfs send has a very significant performance impact on the underlying system - which is very, very odd; 30MB/sec isn't even a tiny fraction of the throughput that drive is capable of, and being an SSD, it isn't really subject to degradation with a little extra IOPS concurrency.
2. Precalculation: There's no way that I'm aware of currently to pre-determine the size of an incremental send, so I can't get any kind of predictive progress bar; this is something I SORELY miss from ZFS. It also makes snapshot management more difficult, because AFAICT there's no way to see how much space on disk is referenced solely by a given snapshot.
3. Incremental sends too big?: incremental btrfs send appears to be sending too much data. I have a "test production" system with a couple of Windows 2008 VMs on it, and it takes hourly rolling snapshots, then does an incremental btrfs send to another system from each snapshot to the next periodically. Problem is, EACH hourly snapshot replication is running 6-10GB of data, which seems like far too much. I don't have any particular way to prove it, since I don't know of a great way to actually calculate the number of changed blocks - but the two Windows 2008 VMs have no native pagefile, so they aren't burning data that way, they're each running VirtIO drivers, and the users aren't changing 6-10GB of data per DAY, much less per hour. Finally, the 6-10GB incremental send size doesn't change significantly whether the increment in question is during the middle of the working day, or in the middle of the night when no users are connected (and when it isn't Patch Tuesday, so it's not like jillions of Windows Updates are coming in either - not that they constitute 120GB-240GB of data!)
I know that last is maddeningly vague, but FWIW I have 30-ish similar setups on ZFS, operating the same way, each with roughly the same number of users running roughly the same set of applications, and those ZFS incrementals are all very consistent; middle-of-the-night incrementals on ZFS running well under 100MB apiece and total bandwidth for an entire day's incremental replication being well under how much bandwidth btrfs send is eating every hour. =\
I'm on my phone so apologies for top posting but please try btrfs-next, I recently fixed a pretty epic performance problem with send which should help you, I'd like to see how much. Thanks,
Josef
Jim Salter <[email protected]> wrote:
Hi list - I'm having problems with btrfs send in general, and incremental send in particular.
1. Performance: in kernel 3.11, btrfs send would send data at 500+MB/sec from a Samsung 840 series solid state drive. In kernel 3.12 and up, btrfs send will only send 30-ish MB/sec from the same drive - though if you interrupt a btrfs send in progress, it will "catch up" to where it was at 500+ MB/sec. This is pretty weird and frustrating. Even weirder and more frustrating, even at 30-ish MB/sec, a btrfs send has a very significant performance impact on the underlying system - which is very, very odd; 30MB/sec isn't even a tiny fraction of the throughput that drive is capable of, and being an SSD, it isn't really subject to degradation with a little extra IOPS concurrency.
2. Precalculation: There's no way that I'm aware of currently to pre-determine the size of an incremental send, so I can't get any kind of predictive progress bar; this is something I SORELY miss from ZFS. It also makes snapshot management more difficult, because AFAICT there's no way to see how much space on disk is referenced solely by a given snapshot.
3. Incremental sends too big?: incremental btrfs send appears to be sending too much data. I have a "test production" system with a couple of Windows 2008 VMs on it, and it takes hourly rolling snapshots, then does an incremental btrfs send to another system from each snapshot to the next periodically. Problem is, EACH hourly snapshot replication is running 6-10GB of data, which seems like far too much. I don't have any particular way to prove it, since I don't know of a great way to actually calculate the number of changed blocks - but the two Windows 2008 VMs have no native pagefile, so they aren't burning data that way, they're each running VirtIO drivers, and the users aren't changing 6-10GB of data per DAY, much less per hour. Finally, the 6-10GB incremental send size doesn't change significantly whether the increment in question is during the middle of the working day, or in the middle of the night when no users are connected (and when it isn't Patch Tuesday, so it's not like jillions of Windows Updates are coming in either - not that they constitute 120GB-240GB of data!)
I know that last is maddeningly vague, but FWIW I have 30-ish similar
setups on ZFS, operating the same way, each with roughly the same number
of users running roughly the same set of applications, and those ZFS
incrementals are all very consistent; middle-of-the-night incrementals
on ZFS running well under 100MB apiece and total bandwidth for an entire
day's incremental replication being well under how much bandwidth btrfs
send is eating every hour. =\
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at https://urldefense.proofpoint.com/v1/url?u=http://vger.kernel.org/majordomo-info.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=yEXVJ85k3S52RAxVbFHaIbe5eV6dKbkfn%2FXLiZd%2BG8E%3D%0A&s=43ef0c011bfafafb636ec0fa76c0e5076fba95df51b64302e1632478b5880fb4
Having trouble building btrfs-next - getting error ".config not found".
me@locutus:~/git/btrfs-next$ make HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/conf.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/zconf.lex.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf --silentoldconfig Kconfig *** *** Configuration file ".config" not found! *** *** Please run some configurator (e.g. "make oldconfig" or *** "make menuconfig" or "make xconfig"). *** make[2]: *** [silentoldconfig] Error 1 make[1]: *** [silentoldconfig] Error 2 SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_32.h SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_64.h SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_x32.h SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_32.h HOSTCC arch/x86/tools/relocs_32.o HOSTCC arch/x86/tools/relocs_64.o HOSTCC arch/x86/tools/relocs_common.o HOSTLD arch/x86/tools/relocs make: *** No rule to make target `include/config/auto.conf', needed by `include/config/kernel.release'. Stop.
Quoted 59 lines · Josef Bacik
I'm on my phone so apologies for top posting but please try btrfs-next, I recently fixed a pretty epic performance problem with send which should help you, I'd like to see how much. Thanks,
Josef
Jim Salter <[email protected]> wrote:
Hi list - I'm having problems with btrfs send in general, and incremental send in particular.
1. Performance: in kernel 3.11, btrfs send would send data at 500+MB/sec from a Samsung 840 series solid state drive. In kernel 3.12 and up, btrfs send will only send 30-ish MB/sec from the same drive - though if you interrupt a btrfs send in progress, it will "catch up" to where it was at 500+ MB/sec. This is pretty weird and frustrating. Even weirder and more frustrating, even at 30-ish MB/sec, a btrfs send has a very significant performance impact on the underlying system - which is very, very odd; 30MB/sec isn't even a tiny fraction of the throughput that drive is capable of, and being an SSD, it isn't really subject to degradation with a little extra IOPS concurrency.
2. Precalculation: There's no way that I'm aware of currently to pre-determine the size of an incremental send, so I can't get any kind of predictive progress bar; this is something I SORELY miss from ZFS. It also makes snapshot management more difficult, because AFAICT there's no way to see how much space on disk is referenced solely by a given snapshot.
3. Incremental sends too big?: incremental btrfs send appears to be sending too much data. I have a "test production" system with a couple of Windows 2008 VMs on it, and it takes hourly rolling snapshots, then does an incremental btrfs send to another system from each snapshot to the next periodically. Problem is, EACH hourly snapshot replication is running 6-10GB of data, which seems like far too much. I don't have any particular way to prove it, since I don't know of a great way to actually calculate the number of changed blocks - but the two Windows 2008 VMs have no native pagefile, so they aren't burning data that way, they're each running VirtIO drivers, and the users aren't changing 6-10GB of data per DAY, much less per hour. Finally, the 6-10GB incremental send size doesn't change significantly whether the increment in question is during the middle of the working day, or in the middle of the night when no users are connected (and when it isn't Patch Tuesday, so it's not like jillions of Windows Updates are coming in either - not that they constitute 120GB-240GB of data!)
I know that last is maddeningly vague, but FWIW I have 30-ish similar
setups on ZFS, operating the same way, each with roughly the same number
of users running roughly the same set of applications, and those ZFS
incrementals are all very consistent; middle-of-the-night incrementals
on ZFS running well under 100MB apiece and total bandwidth for an entire
day's incremental replication being well under how much bandwidth btrfs
send is eating every hour. =\
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at https://urldefense.proofpoint.com/v1/url?u=http://vger.kernel.org/majordomo-info.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=yEXVJ85k3S52RAxVbFHaIbe5eV6dKbkfn%2FXLiZd%2BG8E%3D%0A&s=43ef0c011bfafafb636ec0fa76c0e5076fba95df51b64302e1632478b5880fb4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 13 lines · Jim Salter
Having trouble building btrfs-next - getting error ".config not found".
me@locutus:~/git/btrfs-next$ make HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/conf.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/zconf.lex.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf --silentoldconfig Kconfig *** *** Configuration file ".config" not found! *** *** Please run some configurator (e.g. "make oldconfig" or *** "make menuconfig" or "make xconfig"). *** make[2]: *** [silentoldconfig] Error 1 make[1]: *** [silentoldconfig] Error 2 SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_32.h SYSHDR
Quoted 9 lines
arch/x86/syscalls/../include/generated/uapi/asm/unistd_64.h SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_x32.h SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_32.h HOSTCC arch/x86/tools/relocs_32.o HOSTCC arch/x86/tools/relocs_64.o HOSTCC arch/x86/tools/relocs_common.o HOSTLD arch/x86/tools/relocs make: *** No rule to make target `include/config/auto.conf', needed by `include/config/kernel.release'. Stop.
cd btrfs-next
cp /boot/config-$(uname -r) .config
make oldconfig
(hold enter until you get a command prompt back)
make -j8 && make modules_install && make install
edit your grub.cfg to make sure it's pointing at the new kernel reboot
Thanks,
Josef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTA2rpAAoJEANb+wAKly3BUYMP+wXgRJ3/Jn+Y5TlievpN9ld1 YyiGVzCSz5nybMaVHN4D39TSO9diANd/24GuIlqmGWva91sNYueJ8NFF4n7aKjqH TiGkqsi5J1m0uP06r+vYFb9oQZyCvs3cmh+zGDkH8+k+DI0IyxqaFTZMjDvvaKY8 tEstHviMd91YvdpD1pH/x9/ubSWed61Dv+mwc3wDg8MrDR2EpoC/6Rrc1nWZsdrm hjp8wwootlf3/9Z7XwbYX0L6Z869iOnY3HF+8a93earqO+dyCFOTyIAUbkjf5q5I ZQBDZmtg6KtpH7QlsgPiONRmHELzAV10/bh+3RaOig0cW0x12/EORCWPcw5ktjJ4 PJK8fHK5BcBM6QXI54y8Z6aH/9urA7+FQ/nuxCW0K+Iz4/BxUmRAl6D078MsaLAy eY9vgBn5hm4jmzKfoCqS7fy5NTxMHGCZSCKCcERPJtu5Adjs2KdQRlyfAav8P+Ul /oYHKpTcis29SGq+IXgd0EsBBoy/3zrF8F3tIqhwkQpiqSQVL0TbKEd4EPdbEZaI VFfMYNirJghljl7es4v5ggtRqk8cbAnGKJsc3kw8P67PaedwYH2ecCeNEk9MbIVx /6SuMq0HDspbPctkFHI4Vtu+dYlnYZtgpONtdrVYKsY/n1Ak3sRCD6hj2fwLJcQl SbBBOA0WIJYeaNfMm2yu
=o1cI
-----END PGP SIGNATURE-----