ENOSPC on file deletion with 3.1.6
Hi,
After upgrading my kernel from 2.6.38 (which has worked fine for months) to 3.1.6, I got ENOSPC on recompiling gcc (even though df says there is 16G free of 50G; this is a raid1 setup, so in fact it's 8 of 25).
After this error, I tried to remove the compilation directory (with "rm -r"):
this also gives ENOSPC. I am trying to work around this by first truncating files using "echo > $file", but this fails for some files, again with ENOSPC. Also, removal of files is very slow even if it succeeds.
Moreover, any write operation on the file system now fails with ENOSPC.
Reverting to my old kernel does not help: it now shows the same problem.
Is this a known issue? Is there a way to make this file system unstuck? (I have backups, but I'd like to preserve snapshot information if possible.) Should I try upgrading to an even newer kernel?
Kind regards,
Arie
Arie Peterson wrote (ao):
Quoted 9 lines
After upgrading my kernel from 2.6.38 (which has worked fine for months) to 3.1.6, I got ENOSPC on recompiling gcc (even though df says there is 16G free of 50G; this is a raid1 setup, so in fact it's 8 of 25).
After this error, I tried to remove the compilation directory (with "rm -r"):
this also gives ENOSPC. I am trying to work around this by first truncating files using "echo > $file", but this fails for some files, again with ENOSPC. Also, removal of files is very slow even if it succeeds.
Quoted 7 lines
Moreover, any write operation on the file system now fails with ENOSPC.
Reverting to my old kernel does not help: it now shows the same problem.
Is this a known issue? Is there a way to make this file system unstuck? (I have backups, but I'd like to preserve snapshot information if possible.) Should I try upgrading to an even newer kernel?
Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
FWIW, I also had a disk full just a few days ago. Removed all snapshots and some big files, but to no avail. Likely the background cleanup took too much time. A reboot fixed this.
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 1 line · On Tuesday 03 January 2012 15:06:43 Sander wrote:
Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
Data, RAID1: total=22.72GB, used=14.73GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=2.25GB, used=1.88GB
Metadata: total=8.00MB, used=0.00
Quoted 3 lines
FWIW, I also had a disk full just a few days ago. Removed all snapshots and some big files, but to no avail. Likely the background cleanup took too much time. A reboot fixed this.
OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid booting will fail if the root file system cannot be written to.
Quoted 1 line
Sander
Thanks,
Arie
Arie Peterson wrote (ao):
Quoted 9 lines · On Tuesday 03 January 2012 15:06:43 Sander wrote:
Quoted 1 line
Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
Data, RAID1: total=22.72GB, used=14.73GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=2.25GB, used=1.88GB
Metadata: total=8.00MB, used=0.00
Hm, not full.
Quoted 6 lines
Quoted 3 lines
FWIW, I also had a disk full just a few days ago. Removed all snapshots and some big files, but to no avail. Likely the background cleanup took too much time. A reboot fixed this.
OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid booting will fail if the root file system cannot be written to.
But you did already reboot as you said the old kernel exposed the same behavior?
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 7 lines · On Tuesday 03 January 2012 15:22:58 Sander wrote:
Hm, not full.
Quoted 2 lines
OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid booting will fail if the root file system cannot be written to.
But you did already reboot as you said the old kernel exposed the same behavior?
You are right; the full history of events was:
(- compile new kernel (3.1.6))
- boot new kernel;
- recompile gcc: problem occurs;
- solve problem by removing compilation directory;
- boot old kernel;
- recompile gcc: problem occurs for this kernel as well.
After trying to remove the problematic files for some time, I took the chance and rebooted. After the reboot, the file system still gave ENOSPC on any write operation. However, it was able to boot anyway, and now the removal of the problematic files went much faster and without new ENOSPC. The compilation directory was completely removed, and immediately afterwards, the file system became writeable again.
Sander, thanks for your help.
I am still curious if this a known problem, and if upgrading to a newer kernel might prevent it from reoccurring. (I was planning to wait for 3.2 to be released and included in Gentoo's repository; maybe I shouldn't wait for this...)
Kind regards,
Arie
Quoted 11 lines · Arie Peterson
Quoted 1 line · On Tuesday 03 January 2012 15:06:43 Sander wrote:
Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
Data, RAID1: total=22.72GB, used=14.73GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=2.25GB, used=1.88GB
Metadata: total=8.00MB, used=0.00
Quoted 6 lines
Quoted 3 lines
FWIW, I also had a disk full just a few days ago. Removed all snapshots and some big files, but to no avail. Likely the background cleanup took too much time. A reboot fixed this.
OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid booting will fail if the root file system cannot be written to.
I'd probably run a "btrfs fi balance /", it should be able to recover the space. I'd typically be a little anxious if it was a large filesystem as it's not interruptable except via power-button (in principle it shouldn't matter, but...), but given that your filesystem is quite small, it shouldn't take more than an hour or so.