4.4.0 - no space left with >1.7 TB free space left
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo 3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.
Why would it fail, if there is so much space left?
2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (28) [1454825174:633585][9105:0x7f2b7e33e700], file:collection-33-7895599108848542105.wt, WT_SESSION.checkpoint: collection-33-7895599108848542105.wt write error: failed to write 4096 bytes at offset 20480: No space left on device 2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (28) [1454825174:648740][9105:0x7f2b7e33e700], checkpoint-server: checkpoint server error: No space left on device 2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (-31804) [1454825174:648766][9105:0x7f2b7e33e700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-02-07T06:06:14.648+0000 I - [thread1] Fatal Assertion 28558 2016-02-07T06:06:14.648+0000 I - [thread1]
***aborting after fassert() failure
2016-02-07T06:06:14.694+0000 I - [WTJournalFlusher] Fatal Assertion 28559 2016-02-07T06:06:14.694+0000 I - [WTJournalFlusher]
***aborting after fassert() failure
2016-02-07T06:06:15.203+0000 F - [WTJournalFlusher] Got signal: 6 (Aborted).
# df -h /srv Filesystem Size Used Avail Use% Mounted on /dev/sda4 2.7T 1.1T 1.7T 39% /srv
# btrfs fi df /srv Data, RAID1: total=1.25TiB, used=1014.01GiB System, RAID1: total=32.00MiB, used=240.00KiB Metadata, RAID1: total=15.00GiB, used=13.13GiB GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /srv
Label: 'btrfs' uuid: 105b2e0c-8af2-45ee-b4c8-14ff0a3ca899
Total devices 2 FS bytes used 1.00TiB
devid 1 size 2.63TiB used 1.26TiB path /dev/sda4
devid 2 size 2.63TiB used 1.26TiB path /dev/sdb4btrfs-progs v4.0.1
Tomasz Chmielewski
http://wpkg.org
Quoted 8 lines · On Mon, 08 Feb 2016 18:22:34 +0900 Tomasz Chmielewski <[email protected]> wrote:
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo 3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.
Hello,
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then you should try the patch https://patchwork.kernel.org/patch/7967161/
(Too bad this was not included into 4.4.1.)
Signature · 2 lines
With respect, Roman
Quoted 8 lines · Roman Mamedov
Quoted 5 lines · On Mon, 08 Feb 2016 18:22:34 +0900 Tomasz Chmielewski <[email protected]> wrote:
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Quoted 8 lines
Quoted 3 lines
Every few days I'm getting "no space left" in a container running mongo 3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.
Hello,
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting?
Not sure.
With the last error, a snapshot was made at around 06:06, while "no space left" was reported on 06:14. Suspiciously close to each other, but still, a few minutes away.
Unfortunately I don't have error log for previous cases.
Quoted 4 lines
If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/(Too bad this was not included into 4.4.1.)
I'll keep an eye on it, thanks.
Tomasz Chmielewski
http://www.ptraveler.com
Quoted 2 lines · On Mon, 08 Feb 2016 21:15:38 +0900 Tomasz Chmielewski <[email protected]> wrote:
With the last error, a snapshot was made at around 06:06 "no space left" was reported on 06:14.
If you mean the log that you have posted in your original message, the ENOSPC happened at 06:06 and 14 seconds, not 06:14.
Signature · 2 lines
With respect, Roman
Quoted 9 lines · Roman Mamedov
Quoted 8 lines
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo 3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.
Quoted 7 lines
Hello,
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then you should try the patch https://patchwork.kernel.org/patch/7967161/
(Too bad this was not included into 4.4.1.)
By the way - was it included in any later kernel? I'm running 4.4.5 on that server, but still hitting the same issue.
Tomasz Chmielewski
http://wpkg.org
Quoted 19 lines · On Fri, 08 Apr 2016 20:36:26 +0900 Tomasz Chmielewski <[email protected]> wrote:
Quoted 17 lines · Roman Mamedov
Quoted 9 lines
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo
3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.Hello,
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/
Quoted 4 lines
Quoted 1 line
(Too bad this was not included into 4.4.1.)
By the way - was it included in any later kernel? I'm running 4.4.5 on that server, but still hitting the same issue.
It's not in 4.4.6 either. I don't know why it doesn't get included, or what we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.
Thanks
Signature · 2 lines
With respect, Roman
Quoted 7 lines · Roman Mamedov
Quoted 7 lines
Quoted 6 lines
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/(Too bad this was not included into 4.4.1.)
Quoted 8 lines
Quoted 2 lines
By the way - was it included in any later kernel? I'm running 4.4.5 on that server, but still hitting the same issue.
It's not in 4.4.6 either. I don't know why it doesn't get included, or what we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.
Does it mean 4.5 also doesn't have it yet?
Tomasz Chmielewski
http://wpkg.org
Roman Mamedov posted on Fri, 08 Apr 2016 16:53:32 +0500 as excerpted:
Quoted 4 lines
It's not in 4.4.6 either. I don't know why it doesn't get included, or what we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.
Patches must make it to the current development kernel before they're eligible for stable. Additionally, they need to be cced to stable as well, in ordered to be queued there.
So check 4.5 and 4.6-rc. If it's in neither of those, it's not going to be in stable yet. Once it's in the development kernel, see if it was cced to stable and if needed, ask the author and btrfs devs to cc it to stable.
Tho sometimes stable can get a backlog as well. I know earlier this year they were dealing with one, but I follow release or development, not stable, and don't know what stable's current status is.
If it gets to stable, and it wasn't for a bug introduced /after/ 4.4, it should eventually get into 4.4, as that's an LTS kernel. But it might take awhile, as the above discussion hints.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 7 lines · Roman Mamedov
Quoted 7 lines
Quoted 6 lines
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/(Too bad this was not included into 4.4.1.)
Quoted 8 lines
Quoted 2 lines
By the way - was it included in any later kernel? I'm running 4.4.5 on that server, but still hitting the same issue.
It's not in 4.4.6 either. I don't know why it doesn't get included, or what we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.
FYI, I'm still getting this with 4.5.3, which probably means the fix was not yet included ("No space left" at snapshot time):
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG:
could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No
space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG:
could not close temporary statistics file "pg_stat_tmp/global.tmp": No
space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG:
could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No
space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG:
could not close temporary statistics file "pg_stat_tmp/global.tmp": No
space left on device
I've tried mounting with space_cache=v2, but it didn't help.
Tomasz Chmielewski
http://wpkg.org
Quoted 17 lines · Tomasz Chmielewski
FYI, I'm still getting this with 4.5.3, which probably means the fix was not yet included ("No space left" at snapshot time):
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device /var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
Quoted 1 line
I've tried mounting with space_cache=v2, but it didn't help.
On the good side, I see it's in 4.6-rc7.
Tomasz Chmielewski
http://wpkg.org
Quoted 24 lines · On Fri, 8 Apr 2016 16:53:32 +0500 Roman Mamedov <[email protected]> wrote:
Quoted 21 lines · On Fri, 08 Apr 2016 20:36:26 +0900 Tomasz Chmielewski <[email protected]> wrote:
Quoted 18 lines · Roman Mamedov
Quoted 9 lines
Linux 4.4.0 - btrfs is mainly used to host lots of test containers, often snapshots, and at times, there is heavy IO in many of them for extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo
3.2.1 database. Interestingly, haven't seen this issue in containers with MySQL. All databases have chattr +C set on their directories.Hello,
Do you snapshot the parent subvolume which holds the databases? Can you correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/(Too bad this was not included into 4.4.1.)
Quoted 8 lines
Quoted 2 lines
By the way - was it included in any later kernel? I'm running 4.4.5 on that server, but still hitting the same issue.
It's not in 4.4.6 either. I don't know why it doesn't get included, or what we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.
This got applied now in 4.4.21, thanks.
Signature · 2 lines
With respect, Roman