Skip to contentSkip to conversation

ENOSPC with 270GiB free

DISCUSSION Goswin von Brederlow 4 participants 7 messages Feb 16, 2014 – Feb 18, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty of space left:

% df -m /mnt/nas3
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/nas3-a 19077220 18805132 270773 99% /mnt/nas3

% btrfs fi show
Label: none uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
Total devices 1 FS bytes used 17.91TiB
devid 1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df
Data, single: total=17.89TiB, used=17.88TiB
System, DUP: total=32.00MiB, used=1.92MiB
Metadata, DUP: total=25.50GiB, used=24.89GiB

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

MfG
Goswin

#1 lore raw

Goswin von Brederlow posted on Sun, 16 Feb 2014 14:58:08 +0100 as excerpted:

Quoted 4 lines

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

I saw this on a much (much) smaller filesystem a few weeks ago, when I redid my /boot. In my case it was under a gig total, so mixed-mode, but copying files over in a particular order errored some of them out with ENOSPC. But the way I was copying (using mc) left the ones that hadn't copied selected, and I tried a copy of them again, and/or used mc's directory-diff to find the missing files and copy them over again. After about three times, they all copied.

So some combination of size and metadata wasn't triggering a new block allocation, but coming in a different order, it triggered fine. Again, this was mixed-mode, so data/metadata blocks mixed, and it didn't matter which ran out first since they were combined.

I wonder if you're running into something similar. Can you try doing the copy in a different order, or is it one big file?

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#2 lore raw
Quoted 16 lines · Duncan

Goswin von Brederlow posted on Sun, 16 Feb 2014 14:58:08 +0100 as excerpted:

Quoted 4 lines

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

I saw this on a much (much) smaller filesystem a few weeks ago, when I redid my /boot. In my case it was under a gig total, so mixed-mode, but copying files over in a particular order errored some of them out with ENOSPC. But the way I was copying (using mc) left the ones that hadn't copied selected, and I tried a copy of them again, and/or used mc's directory-diff to find the missing files and copy them over again. After about three times, they all copied.

Quoted 8 lines

So some combination of size and metadata wasn't triggering a new block allocation, but coming in a different order, it triggered fine. Again, this was mixed-mode, so data/metadata blocks mixed, and it didn't matter which ran out first since they were combined.

I wonder if you're running into something similar. Can you try doing the copy in a different order, or is it one big file?

I'm using rsync and towards the last few GB before it gives ENOSPC the filesystem gets realy slow and eats more and more cpu time. I'm copying multi gigabyte files and the second last file managed 2MB/s and the failed one came down to ~100K/s at the end. So trying a lot of different files isn't realy feasable, timewise.

MfG
Goswin

#3 lore raw
In reply to Goswin von Brederlow

Did you already try this?? [1]:

btrfs fi balance start -dusage=5 /mnt/nas3

Cheers, dan

[1] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space

Quoted 32 lines · Goswin von Brederlow

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty of space left:

% df -m /mnt/nas3
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/nas3-a 19077220 18805132 270773 99% /mnt/nas3

% btrfs fi show
Label: none uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
Total devices 1 FS bytes used 17.91TiB
devid 1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df
Data, single: total=17.89TiB, used=17.88TiB
System, DUP: total=32.00MiB, used=1.92MiB
Metadata, DUP: total=25.50GiB, used=24.89GiB

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 37 lines · Dan van der Ster

Did you already try this?? [1]:

btrfs fi balance start -dusage=5 /mnt/nas3

Cheers, dan

[1] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space

Quoted 28 lines · Goswin von Brederlow

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty of space left:

% df -m /mnt/nas3
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/nas3-a 19077220 18805132 270773 99% /mnt/nas3

% btrfs fi show
Label: none uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
Total devices 1 FS bytes used 17.91TiB
devid 1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df
Data, single: total=17.89TiB, used=17.88TiB
System, DUP: total=32.00MiB, used=1.92MiB
Metadata, DUP: total=25.50GiB, used=24.89GiB

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

MfG
Goswin

I did and that isn't the problem. Balancing only frees up partially used block groups so they can be reused. But the problem is that the remaining free block groups are not getting used.

MfG
Goswin

#5 lore raw
In reply to Goswin von Brederlow

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 20 lines · Goswin von Brederlow

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty of space left:

% df -m /mnt/nas3 Filesystem 1M-blocks Used Available
Use% Mounted on /dev/mapper/nas3-a 19077220 18805132 270773
99% /mnt/nas3

% btrfs fi show Label: none uuid:
4b18f84e-2499-41ca-81ff-fe1783c11491 Total devices 1 FS bytes used
17.91TiB devid 1 size 18.19TiB used 17.94TiB path
/dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df Data, single: total=17.89TiB, used=17.88TiB System,
DUP: total=32.00MiB, used=1.92MiB Metadata, DUP: total=25.50GiB, used=24.89GiB

Quoted 7 lines

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

What kernel? Can you give btrfs-next a try? Mount with -o enospc_debug and when you get enospc send the dmesg. Thanks,

Josef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTA2bsAAoJEANb+wAKly3BLB4P/0W5RCk9YZQ+WJcDs8/g82Be G5oWx+0N2eYKIttCWxrvXlx2mpVP5mWimzYqs+V7SvaVgbm3KlwuID9JQpuOQ1lH ihnxr48kdwNa8J159oEU7oVRJ84D1b0D21MZzIKU9YoNoDHTQQsELe4aojniKMIY 4RFh/8qWjMm2BGKW4jZwKzIHoF9rt+kU7Lhfm9XOi0OCut8FlFSjSwsiaN8NFJAf DFtkgfZTRJUxiq3dXn7m6yzspS3L5atAfd862UbFDrA7EcYIZnc+zw9/hUxX8JwP N3espmJQ03DRn8LLoOZtBMZ+NUnRgoOd33k1z9jeB7RnqN165vSQaYibUw6Zk+Vh qADBAy+f8aHtBF8yNIuwqEXH9D5fXlfgYUnWW/YBiw6+dbguxb4CKIzkdh67paah zvpLrXkZd95taqXFkaj0PZydwSVkSIbrPuNllRku02Bz8XxPjE0rlW+wCr2uHI2y Ne2jpGdCClJCfLTZC8SsHaLeYktvEVBnQ6ZOriU+Rl1hzvi+ieG7leRHIa1K7VED 669zuZ4QfYvdU7lnXzTuC36NmmaSC4wrLaYTPAytU4OLB5dIby9nyF+25t60hq5n pb2R5tn7heH9928aVktFMTuAWUBTI7B1O1ZGB/cTcYKduYXOD9Y5ljEalA7HhUpb PwuiepAP1uBkiBPQDYH1
=tCLT
-----END PGP SIGNATURE-----

#6 lore raw
Quoted 32 lines · Josef Bacik

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 20 lines · Goswin von Brederlow

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty of space left:

% df -m /mnt/nas3 Filesystem 1M-blocks Used Available
Use% Mounted on /dev/mapper/nas3-a 19077220 18805132 270773
99% /mnt/nas3

% btrfs fi show Label: none uuid:
4b18f84e-2499-41ca-81ff-fe1783c11491 Total devices 1 FS bytes used
17.91TiB devid 1 size 18.19TiB used 17.94TiB path
/dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df Data, single: total=17.89TiB, used=17.88TiB System,
DUP: total=32.00MiB, used=1.92MiB Metadata, DUP: total=25.50GiB, used=24.89GiB

Quoted 7 lines

As you can see there are still 270GiB free and plenty of block groups free on the device too.

So why isn't btrfs allocating a new block group to store more data?

Quoted 4 lines

What kernel? Can you give btrfs-next a try? Mount with -o enospc_debug and when you get enospc send the dmesg. Thanks,

Josef

Standard Debian linux kernel:

Linux nas3 3.12-1-amd64 #1 SMP Debian 3.12.9-1 (2014-02-01) x86_64 GNU/Linux

Compiling new kernel and rebooting will take some time. But it looks like I can remount with enospc_debug. I will try if that outputs anything usefull first. So far I got:

[258988.006643] btrfs: disk space caching is enabled

MfG
Goswin