2.6.33.1: ext4 disk free weirdness

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as ext4. /mnt filesystem is only being added to and currently large files (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3 I could not duplicate the issue and the amount of disk used stabilised at 182G. The df results, whilst /mnt is mounted as ext4, are below:

1 [29/03 15:05:58] >> df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  317G  555G  37% /mnt
1 [29/03 15:11:05] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  224G  647G  26% /mnt
1 [29/03 15:41:38] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  670G  201G  77% /mnt 
1 [29/03 15:48:32] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  246G  626G  29% /mnt 
1 [29/03 15:48:36] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  313G  559G  36% /mnt 
1 [29/03 15:48:37] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  396G  475G  46% /mnt 
1 [29/03 15:48:37] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  615G  256G  71% /mnt 
1 [29/03 15:48:39] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  671G  201G  77% /mnt 
1 [29/03 15:48:40] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  671G  201G  77% /mnt 
1 [29/03 15:48:40] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  215G  656G  25% /mnt 
1 [29/03 15:48:41] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  334G  538G  39% /mnt 
1 [29/03 15:48:42] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  522G  349G  60% /mnt 

1 [29/03 15:48:43] >>

Quoted 6 lines · CaT

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as ext4. /mnt filesystem is only being added to and currently large files (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3 I could not duplicate the issue and the amount of disk used stabilised at 182G. The df results, whilst /mnt is mounted as ext4, are below:

I would say this is likely speculative allocation due to delalloc (ext4 has to reserve worst-case metadata amounts to prepare for delalloc writeback). When the file data actually gets written out, the worst-case reservation is freed up again. We also flush and switch to nodelalloc when the filesystem is pretty close to full. see ext4_nonda_switch() for example.

This looks pretty severe though, more than I would expect from that behavior.

Quoted 14 lines · Eric Sandeen

Quoted 6 lines · CaT

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as ext4. /mnt filesystem is only being added to and currently large files (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3 I could not duplicate the issue and the amount of disk used stabilised at 182G. The df results, whilst /mnt is mounted as ext4, are below:

I would say this is likely speculative allocation due to delalloc (ext4 has to reserve worst-case metadata amounts to prepare for delalloc writeback). When the file data actually gets written out, the worst-case reservation is freed up again. We also flush and switch to nodelalloc when the filesystem is pretty close to full. see ext4_nonda_switch() for example.

Quoted 2 lines

This looks pretty severe though, more than I would expect from that behavior.

This is fixed in commit d330a5bef, which which got merged post 2.6.34-rc2, and which we need to get into 2.6.33 stable. If you cherry-pick that commit, I think you'll see that it fixes this problem.

- Ted

Quoted 15 lines · [email protected] wrote:

Quoted 14 lines · Eric Sandeen

Quoted 6 lines · CaT

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as ext4. /mnt filesystem is only being added to and currently large files (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3 I could not duplicate the issue and the amount of disk used stabilised at 182G. The df results, whilst /mnt is mounted as ext4, are below:

I would say this is likely speculative allocation due to delalloc (ext4 has to reserve worst-case metadata amounts to prepare for delalloc writeback).  When the file data actually gets written out, the worst-case reservation is freed up again.  We also flush and switch to nodelalloc when the filesystem is pretty close to full. see ext4_nonda_switch() for example.

Quoted 7 lines

Quoted 2 lines

This looks pretty severe though, more than I would expect from that behavior.

This is fixed in commit d330a5bef, which which got merged post 2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you cherry-pick that commit, I think you'll see that it fixes this problem.

Oh, right, spaced that one.  Thanks for the reminder Ted.  :)

And from now on ext3-mounted-as-ext4 is special-cased so you won't get delalloc, and you won't get this behavior at all (nor will you get any delalloc benefits, FWIW).

Quoted 24 lines · Eric Sandeen

Quoted 15 lines · [email protected] wrote:

Quoted 14 lines · Eric Sandeen

Quoted 6 lines · CaT

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as ext4. /mnt filesystem is only being added to and currently large files (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3 I could not duplicate the issue and the amount of disk used stabilised at 182G. The df results, whilst /mnt is mounted as ext4, are below:

I would say this is likely speculative allocation due to delalloc (ext4 has to reserve worst-case metadata amounts to prepare for delalloc writeback). When the file data actually gets written out, the worst-case reservation is freed up again. We also flush and switch to nodelalloc when the filesystem is pretty close to full. see ext4_nonda_switch() for example.

Quoted 6 lines

Quoted 2 lines

This looks pretty severe though, more than I would expect from that behavior.

This is fixed in commit d330a5bef, which which got merged post 2.6.34-rc2, and which we need to get into 2.6.33 stable. If you cherry-pick that commit, I think you'll see that it fixes this problem.

Oh, right, spaced that one. Thanks for the reminder Ted. :)

Quoted 4 lines

And from now on ext3-mounted-as-ext4 is special-cased so you won't get delalloc, and you won't get this behavior at all (nor will you get any delalloc benefits, FWIW).

Out of curiosity, when you say ext3-mounted-as-ext4, are you referring to filesystems without ext4 features mounted -t ext4, or filesystems mounted -t ext3 when CONFIG_EXT4_USE_FOR_EXT23 is defined?

-Justin

This is fixed in commit d330a5bef, which which got merged post 2.6.34-rc2, and which we need to get into 2.6.33 stable. If you cherry-pick that commit, I think you'll see that it fixes this problem.