Skip to contentSkip to conversation

kernel 4.1: no space left on device

DISCUSSION Stefan Priebe 4 participants 7 messages Jun 27, 2015 – Jul 18, 2015 1 message not in the archive Btrfs file system

lore
Share & tools
lore

Stefan Priebe

author
#0 lore raw

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

Greets,
Stefan

#1 lore raw
Quoted 16 lines · Stefan Priebe

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

Greets,
Stefan

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Assuming you ruled out accumulation of snapshots?

Signature · 7 lines
--Moby


First they came for the Jews and I did not speak out because I was not a Jew.
Then they came for the Communists and I did not speak out because I was not a Communist.
Then they came for the trade unionists and I did not speak out because I was not a trade unionist.
Then they came for me and there was no one left to speak out for me.  --  Pastor Martin Niemöller
#2 lore raw
In reply to Stefan Priebe
Quoted 9 lines · On Sat, 27 Jun 2015 17:32:04 +0200 Stefan Priebe <[email protected]> wrote:

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for both devices.

Signature · 2 lines
With respect,
Roman
author
#3 lore raw

Am 27.06.2015 um 17:51 schrieb Roman Mamedov:

Quoted 7 lines
Quoted 4 lines · On Sat, 27 Jun 2015 17:32:04 +0200 Stefan Priebe <[email protected]> wrote:

Hi,

while having some big btrfs volumes (44TB + 37TB).

Quoted 7 lines
Quoted 5 lines

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for both devices.

sorry - here for one server:
[ ~]# btrfs filesystem df /vmbackup/
Data, single: total=7.58TiB, used=7.58TiB System, DUP: total=8.00MiB, used=1.19MiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=69.00GiB, used=38.25GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=3.30MiB

[ ~]# btrfs fi show /vmbackup/
Label: none uuid: 0df5ed87-07da-4c15-9a2b-1d0ff6ec208d
Total devices 1 FS bytes used 7.62TiB
devid 1 size 43.66TiB used 7.72TiB path /dev/md50

btrfs-progs v4.0.1

Stefan

author
#5 lore raw
In reply to a message not in the archive

Am 27.06.2015 um 23:50 schrieb Ruslanas Gžibovskis:

Quoted 4 lines

Hi,

df -i ?
Maybe inode?

df -i for btrfs?

# df -i |grep vmbackup
/dev/md50              0      0        0     - /vmbackup
Quoted 50 lines

Have a nice $day_time.

On Sat, 27 Jun 2015 19:11 Stefan Priebe <[email protected] <mailto:[email protected]>> wrote:

    Am 27.06.2015 um 17:51 schrieb Roman Mamedov:
     > On Sat, 27 Jun 2015 17:32:04 +0200
     > Stefan Priebe <[email protected]
    <mailto:[email protected]>> wrote:
     >
     >> Hi,
     >>
     >> while having some big btrfs volumes (44TB + 37TB).
     >>
     >> I see on a regular basis the "no space left on device" message.
    I'm only
     >> able to "fix" this. By running btrfs balance AND unmounting and
     >> remounting the btrfs volume.
     >>
     >> Is there any way to debug / workaround this one?
     >
     > You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for
    both devices.

    sorry - here for one server:
    [ ~]# btrfs filesystem df /vmbackup/
    Data, single: total=7.58TiB, used=7.58TiB
    System, DUP: total=8.00MiB, used=1.19MiB
    System, single: total=4.00MiB, used=0.00B
    Metadata, DUP: total=69.00GiB, used=38.25GiB
    Metadata, single: total=8.00MiB, used=0.00B
    GlobalReserve, single: total=512.00MiB, used=3.30MiB

    [ ~]# btrfs fi show /vmbackup/
    Label: none  uuid: 0df5ed87-07da-4c15-9a2b-1d0ff6ec208d
              Total devices 1 FS bytes used 7.62TiB
              devid    1 size 43.66TiB used 7.72TiB path /dev/md50

    btrfs-progs v4.0.1

    Stefan
    --
    To unsubscribe from this list: send the line "unsubscribe
    linux-btrfs" in
    the body of a message to [email protected]
    <mailto:[email protected]>
    More majordomo info at http://vger.kernel.org/majordomo-info.html
#6 lore raw
In reply to Stefan Priebe
Quoted 9 lines · Stefan Priebe

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

What kernel version?
Have you tried to reproduce with a newer kernel and an older kernel? I suggest trying 4.1 for newer, and at oldest 3.18.16 for regression testing.
You see the message where/when?
And what's the workload happening at the time?
If you pause/stop that work load, can you reproduce it by by cp'ing
some files? Does it happen with small files (less than 16KB) or medium (100MB) or large (1+GB) files, or always?
About how many snapshots or reflink'd large files?

If it were me, after the above upgrade and regression testing, I'd do the following and add all information thus far (version of kernel, and btrfs-progs, and the workflow description) to a bugzilla.kernel.org bug and *attach* the long output results as files to that bug:

1. btrfs check, without --repair, attach complete output to bug
2. mount with option enospc_debug
3. strace cp to reproduce the problem, attach output to bug
4. dmesg from the time of mount at #2, through the end of the strace in 3, attach output to bug
5. remount with option clear_cache and see if you can reproduce the problem

Signature · 1 line
Chris Murphy
author
#7 lore raw
In reply to Stefan Priebe

Still nobody? Upgraded to 4.2-rc2 and i still see the out of space situation on two 30TB und 40TB arrays every week.

Stefan

Am 27.06.2015 um 17:32 schrieb Stefan Priebe:

Quoted 12 lines

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only able to "fix" this. By running btrfs balance AND unmounting and remounting the btrfs volume.

Is there any way to debug / workaround this one?

Greets,
Stefan