Weird behavior (no commit?) when writing to a really slow block device
Hello,
I am rsyncing some files from fast devices (100-200 MB/s) to a slow remote one (15 MB/s) over nbd.
There is some strange behavior, at least on my current kernel 5.10.161:
* free space shown in "df" does NOT decrease even after many hours and
hundreds of GBs already written;
* freshly written files (already fully copied by rsync log) are shown with their real size in "ls -la", but all have zero size in "du".
I keep an eye on: watch "grep -e Dirty -e Writeback: /proc/meminfo" This currently shows:
Dirty: 554936 kB
Writeback: 3388616 kB
The latter figure hovers around 3.5-4 GB all the time. For these syncing setups, I have these settings tweaked on this machine (so I guess it's not as large as it could have been with 64 GB of RAM):
vm.dirty_background_ratio=5
vm.dirty_ratio=10
As I understand the described weird symptoms would have cleared up if there was a complete flush of data and metadata. And I suspect issuing a "sync" on command-line would cause that to occur. But the question is why this didn't occur on its own, as said above, over many hours of copying? If I disconnect the block device now, wouldn't everything copied so far be actually lost?
I know there are the "commit" and "flushoncommit" options, but in this case I haven't overridden either. Mount options are:
rw,noatime,nodiratime,compress=zstd:3,nossd,space_cache=v2,skip_balance,subvolid=5,subvol=/
So if I had to guess, given a slow block device and the constant ongoing write load trying to stream at a much faster rate, the "commit" doesn't actually happen, neither in its predefined default interval of 30 seconds, nor during much much longer (hours).
Signature · 2 lines
With respect, Roman
Quoted 2 lines · On Sun, 8 Jan 2023 02:03:31 +0500 Roman Mamedov <[email protected]> wrote:
if there was a complete flush of data and metadata. And I suspect issuing a "sync" on command-line would cause that to occur.
In fact, no. :D
Running a "sync" simply does not return while the rsync process is still actively copying data to the slow device.
It's stuck for 30 minutes by now, despite it would take less than 4 minutes to write out the 3.5 GB of the writeback buffer at 15 MB/sec of the target device.
And the free space in "df" still did not update from what it was when I started the copy process 3 hours ago. rsync has managed to write 185 GB since then, according to its /proc/xxx/io.
Signature · 2 lines
With respect, Roman