New: free space lossage on busy system with bigalloc enabled and 128KB cluster

https://bugzilla.kernel.org/show_bug.cgi?id=151491

            Bug ID: 151491
           Summary: free space lossage on busy system with bigalloc
                    enabled and 128KB cluster
           Product: File System
           Version: 2.5
    Kernel Version: 4.1.8
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 227581
--> https://bugzilla.kernel.org/attachment.cgi?id=227581&action=edit
details of fault with scripts for reproduction

file system with bigalloc enabled and 128KB cluster size with a large number of 2MB files being created/overwritten/deleted loses usable space.

Running du and df gives wildly different usage with df showing much more usage than du. lsof shows no phantom open files. Using dd to fill the file system shows that df's version of free space is operative, but unmounting and remounting the file system returns the free space. There is no difference between df and du usage after remount.

The fault does not seem to be present in the 4.7 kernel (or it takes a lot more activity for it to show up).

I will build 4.4.16 and retest to see if is present there.

We do have a(n obnoxious) workaround of periodically unmounting/remounting files systems.

Details of configurations and tests in the attached document

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #1 from Matthew L. Martin <[email protected]> --- The fault is in linux 4.4.16. After running the populate script for a few hours du and df disagree a great deal:

# df -h /mnt/hdd_sd[gh]; du -hs  /mnt/hdd_sd[gh]
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg        5.5T  129G  5.3T   3% /mnt/hdd_sdg
/dev/sdh        5.5T   20G  5.4T   1% /mnt/hdd_sdh
20G    /mnt/hdd_sdg
20G    /mnt/hdd_sdh

# lsof | grep -e sdg -e sdh
jbd2/sdg- 32609           root  cwd       DIR              9,127      4096     
  192 /
jbd2/sdg- 32609           root  rtd       DIR              9,127      4096     
  192 /
jbd2/sdg- 32609           root  txt   unknown                                  
      /proc/32609/exe
jbd2/sdh- 32614           root  cwd       DIR              9,127      4096     
  192 /
jbd2/sdh- 32614           root  rtd       DIR              9,127      4096     
  192 /
jbd2/sdh- 32614           root  txt   unknown                                  
      /proc/32614/exe

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #2 from Matthew L. Martin <[email protected]> --- I believe that I have confirmed that this fault is not present in linux 4.7. After running the reproduction script for over three hours I have not seen a difference between the usage reported by du and df:

# df -h /mnt/hdd_sd[gh]; du -hs  /mnt/hdd_sd[gh]
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg        5.5T   20G  5.4T   1% /mnt/hdd_sdg
/dev/sdh        5.5T   20G  5.4T   1% /mnt/hdd_sdh
20G    /mnt/hdd_sdg
20G    /mnt/hdd_sdh

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #3 from Matthew L. Martin <[email protected]> --- Unfortunately, after an extended run of the test scripts the fault presented itself in the 4.7 kernel:

[root@d-ceph01 ~]# df -h /mnt/hdd_sd[gh]; du -hs /mnt/hdd_sd[gh]

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg        5.5T  669G  4.8T  13% /mnt/hdd_sdg
/dev/sdh        5.5T   20G  5.4T   1% /mnt/hdd_sdh

20G /mnt/hdd_sdg
20G /mnt/hdd_sdh

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

Fischreiher ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #4 from Fischreiher ([email protected]) ---
I have a similar or same problem on a Linux based Enigma2 set-top box

- ext4
- kernel 4.8.3
- bigalloc enabled
- cluster size of 262144

In normal use of the set-top box, the free space lossage is 10s of Gigabytes per day. The problem can be easily reproduced:

When creating a fresh file, there is a significant difference in file size (ls -la) and disk usage (du). When making two copies of the file ..

gbquad:/hdd/test# cp file file.copy1
gbquad:/hdd/test# cp file file.copy2
gbquad:/hdd/test# ls -la
-rw-------    1 root     root     581821460 Nov  1 18:52 file
-rw-------    1 root     root     581821460 Nov  1 18:56 file.copy1
-rw-------    1 root     root     581821460 Nov  1 18:57 file.copy2
gbquad:/hdd/test# du *
607232  file
658176  file.copy1
644864  file.copy2

... all three files show an overhead in the ~10% range, and the overhead is different for these files although their md5sums are equal.

When deleting a file (rm), the overhead remains occupied on the disk. For example, after deleting "file", "df" reports approx. 581821460 more bytes free, not 607232 kbytes more free space. The overhead (607232 kB - 581821460 B = approx. 39 MB) remains blocked.

When unmounting and mounting again, the blocked space becomes free again, and in addition the overhead of those files that were not deleted also disappears, so that after a re-mount the 'file size' and 'disk usage' match for all files (except for rounding up to some block size).

I found that
echo 3 > /proc/sys/vm/drop_caches
seems to detach the blocked disk space from the files (so that 'du file' no longer includes the offset), but it does not free the space, 'df' still shows all file overheads as used disk space.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #5 from Theodore Tso ([email protected]) ---
Can you try replicating this on an upstream kernel, running in a controlled environment (e.g., using kvm) and then give us reliable reproduction instructions --- e.g., using simple shell scripts, and which doesn't depend on the vagaries of the settop box software, and random versions of du, df, etc.

For bonus points, use get a copy of kvm-xfstests[1][2], and run the test using
scripts cut and pasted into "kvm-xfstests shell". That way we will be able to
reproduce *exactly* what you are doing.

[1] https://github.com/tytso/xfstests-bld
[2]
https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-xfstests.md

Thanks!!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

Betacentauri ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #6 from Betacentauri ([email protected]) --- First result:
When I revert this commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/ext4?h=v4.1&id=9d21c9fa2cc24e2a195a79c27b6550e1a96051a4
in a 4.1 ARM kernel the problem doesn't occur any longer. Scripts for replicating in the kvm-xfstests environment will follow later.

By the way: The setup boxes fischreiher and me are talking about use only slightly adapted upstream kernel to support broadcom hardware. fs directory in kernel sources is not changed.

Frank

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #7 from Betacentauri ([email protected]) --- Created attachment 260631
--> https://bugzilla.kernel.org/attachment.cgi?id=260631&action=edit
Test script

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #8 from Betacentauri ([email protected]) --- Created attachment 260633
--> https://bugzilla.kernel.org/attachment.cgi?id=260633&action=edit
Test script output

I have used a 4.9 32 bit kernel (config from kernel-configs folder).

Output shows how used space in df command increases. Also files generated by dd and cp have different file size in ls command.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #9 from Betacentauri ([email protected]) --- Forgot to say that the script is for the kvm-xfstests environment. And the output already shows the output of that environment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

Eric Whitney ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #10 from Eric Whitney ([email protected]) ---
I've been able to reproduce the reported problem on my test system running a 4.14 x86-64 kernel with the supplied test script. Thanks for supplying it!

The block reporting errors from du and df are likely caused by delayed allocation accounting bugs. Experiments with an instrumented kernel show that the number of delayed allocated blocks is occasionally overcounted as the test files are physically allocated, leaving a residual value behind once allocation is complete. This residual value remains once a file has been fully written out or deleted, and distorts the results reported by du or df. Interestingly, the overcounting isn't deterministic and varies from run to run.

Part of the overcounting appears due to code in ext4_ext_map_blocks() that increases i_reserved_data_blocks when new clusters are allocated. This code has been previously implicated in other observed failures and in this case appears to contribute some but not always all of the overcounted clusters seen when running the test script. Kernel traces indicate that there is usually another as yet unknown contributor to the overcount.

Ted has suggested a temporary workaround which can be used to avoid the reported problems, though it may have a significant workload-dependent performance impact. Delayed allocation can simply be disabled by using the nodelalloc mount option. I've tested this with repeated runs of the supplied test script, and it avoids the reported problems as expected.

Reverting "ext4: don't release reserved space for previously allocated cluster" (9d21c9fs2cc2) isn't an attractive option because doing so would expose users to potential data loss. The purpose of the patch was to fix cases where the number of outstanding delayed allocation blocks were undercounted. Undercounting can lead to unexpected free space exhaustion at writeback time, among other things.

I'll see what more I can learn from some additional experimentation.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #11 from Betacentauri ([email protected]) --- nodelalloc mount options workarounds the problem in the test environment. But I also checked with real ARM system with 4.1.37 kernel:

root@sf4008:/media# mount 
...
/dev/sda on /media/sda type ext4 (rw,relatime,nodelalloc,data=ordered)
root@sf4008:/media# ls -las sda/testfiles/
 10240 -rw-r--r--    1 root     root      10485760 Nov 17 16:47 test
 10304 -rw-r--r--    1 root     root      10485760 Nov 17 16:47 test1

Test is a little bit different. Only 10 MB files are generated. In most cases file size (first column) is equal, but in some cases file size still differs like in above example. It's not deterministic for me when it happens. But I only see 2 sizes. 10240 or 10304. With delalloc the file size was much more random.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #12 from Eric Whitney ([email protected]) ---
After many attempts, I'm unable to reproduce the newly reported behavior for the delalloc workaround in comment 11 on my x86-64 test system (which is not an xfstests-bld test appliance) running either a current 4.14 kernel or an older Debian Jessie 4.8 kernel. I consistently get a reported value of 10240 1k units, which is correct for the reported size.

However, in the process of running my trials I arrived at a simpler reproducer that should be helpful in identifying the source of the original space reporting problem. There's no need to copy the first test file if the test system's free memory is sufficiently limited relative to the size of the test file - a simple sequential write of a single test file suffices. In fact, the tighter the free memory, the more likely the problem occurs and the likelihood of larger reporting errors increases. A test system with ample free memory won't exhibit the problem at all.

I'm getting workable kernel traces with the simpler reproducer, and the free memory-related behavior suggests a direction, so I'll see where that takes me.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #13 from Betacentauri ([email protected]) ---
The ARM machine I use has very little free memory. So that fits to your analysis.
With your information I could also reproduce it in the xfstests environment with nodelalloc. But only 2 times. I set memory in the virtual machine to 256MB (in config.kvm). Then I mounted the filesystem with nodelalloc and executed this little script several times:

#!/bin/bash

i=0
while [ $i -lt 10 ]; do
dd if=/dev/zero of=./test$i bs=1M count=200 > /dev/null cp test$i testx_$i
sync
let i=i+1
echo $i
done

Result was this:

root@kvm-xfstests:/media/test# ls -las
total 4097028
   256 drwxr-xr-x 3 root root      4096 Nov 20 17:43 .
     4 drwxr-xr-x 3 root root      4096 Nov 20 17:33 ..
   256 -rwxr-xr-x 1 root root       155 Nov 20 17:48 h.sh
   256 drwx------ 2 root root     16384 Nov 20 17:26 lost+found
204800 -rw-r--r-- 1 root root 209715200 Nov 20 17:48 test0
205056 -rw-r--r-- 1 root root 209715200 Nov 20 17:48 test1
204800 -rw-r--r-- 1 root root 209715200 Nov 20 17:48 test2
204800 -rw-r--r-- 1 root root 209715200 Nov 20 17:48 test3
204800 -rw-r--r-- 1 root root 209715200 Nov 20 17:48 test4
...

One file has size 205056. But I cannot really reproduce it.

So better focus on fixing the bug than on trying to reproduce it with nodelalloc. If I find a way to reproduce, I'll let you know ;-)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #14 from Eric Whitney ([email protected]) ---
I've been able to identify the unknown contributor to the delalloc accounting error as discussed previously (in comment 10).

When ext4 is writing back a previously delalloc'ed extent that contains just a portion of a cluster, and then delallocs an extent that contains another disjoint portion of that same cluster, the count of delalloc'ed clusters (i_reserved_data_blocks) is incorrectly incremented. The cluster has been physically allocated during writeback, but the subsequent delalloc write does not discover that allocation. This is because the code in ext4_da_map_blocks() checks for a previously physically allocated block at the point of allocation rather than a previously physically allocated cluster spanning the point of allocation.

The effect is to bump the delalloc'ed cluster count for clusters that will never be allocated (since they've already been allocated), and the overcount will therefore never be reduced.

It's more likely this problem would occur when writing files sequentially if the test system was under memory pressure, resulting in writeback activity in parallel with delalloc writes. The magnitude of the overcount is also likely to be larger in this situation. This correlates well with the observation that the reproducer for the accounting errors is more likely to reproduce the problem on a test system with little free memory.

I've been testing a prototype patch that appears to fix this problem. However, I've also identified at least two other unrelated delalloc accounting problems for bigalloc file systems whose effects are masked by the other contributor to overcounting in ext4_ext_map_blocks(). Fixing it results in failures caused by these other problems when running xfstests-bld on bigalloc. So, there's a lot of work yet to be done before it's time to post patches.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #15 from Fischreiher ([email protected]) ---
Thanks a lot for your investigation, Eric, and for describing these details. It is great to know that this issue is in competent hands.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #16 from Fischreiher ([email protected]) ---
Hi Erik, I got the message that this is not an easy nor quick fix, but is it still ob your list?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #17 from Theodore Tso ([email protected]) ---
Eric has still been working on it (he has been reporting on it on our weekly
ext4 concalls, and we've been discussing it). He's identified an approach and
has patches which he is refining, perfecting, and testing. Hopefully there
will be something that can be released for users to test in the near future.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #18 from Fischreiher ([email protected]) --- Great, thank you, I'll be patient.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #19 from Betacentauri ([email protected]) --- Any news regarding this ticket? Is the problem in the meantime fixed?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #20 from Eric Whitney ([email protected]) ---
4.20 contains patches that correct delalloc cluster accounting for bigalloc file systems. They were merged at the beginning of the release cycle. See "ext4: generalize extents status tree search functions" (ad431025aecd) and the following five patches. They should address all the problems described in this bugzilla.

Any independent testing would be appreciated. I'd recommend working with the latest mainline rc available at this time, which is 4.20-rc6.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #21 from Betacentauri ([email protected]) --- Thanks for the patches!

In kvm-xfstests environment with a 4.20-rc6 kernel I cannot reproduce the bug anymore. I also created a tmpfs with several big files to reduce free memory, but still there are no problems :-)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=151491

--- Comment #22 from Eric Whitney ([email protected]) ---
Very good - thanks for the testing! I'll close this bug out at the end of the 4.20 release cycle if no negative test results are reported by then.

You are receiving this mail because:
You are watching the assignee of the bug.