BTRFS claims that empty directory is not empty and refuses to delete it

DISCUSSION Martin Steigerwald 3 participants 5 messages Jul 15, 2014 – Aug 27, 2014 Btrfs file system

Share & tools

Martin Steigerwald

author

2014-07-15 09:09 UTC #0 lore raw

Hello!

This is with 3.16-rc4 – stepped back to this one after having two hangs in one day with 3.16-rc5, see other thread started by me:

martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin 14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1>

Whats this?

I had this weeks ago already and just moved it out of the way at that time, just now stumbled upon it again.

This is the same BTRFS Dual SSD RAID as mentioned in the other post:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 123.20GiB
devid 1 size 160.00GiB used 159.98GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 159.98GiB path /dev/dm-3

Btrfs v3.14.1
merkaba:~#1> btrfs fi df /home
Data, RAID1: total=154.95GiB, used=120.61GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=2.59GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> df -hT /home
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/msata-home btrfs 320G 247G 69G 79% /home

merkaba:~> file -sk /dev/sata/home
/dev/sata/home: symbolic link to `../dm-3'
merkaba:~> file -sk /dev/dm-3
/dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384, leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a, 132303151104/343597383680 bytes used, 2 devices

Ciao,

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Torbjørn

2014-07-15 09:31 UTC #1 lore raw

Quoted 53 lines · Martin Steigerwald

Hello!
This is with 3.16-rc4 – stepped back to this one after having two hangs in one day with 3.16-rc5, see other thread started by me:
martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin 14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1>
Whats this?
I had this weeks ago already and just moved it out of the way at that time, just now stumbled upon it again.
This is the same BTRFS Dual SSD RAID as mentioned in the other post:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 123.20GiB
devid 1 size 160.00GiB used 159.98GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 159.98GiB path /dev/dm-3
Btrfs v3.14.1
merkaba:~#1> btrfs fi df /home
Data, RAID1: total=154.95GiB, used=120.61GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=2.59GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> df -hT /home
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/msata-home btrfs 320G 247G 69G 79% /home
merkaba:~> file -sk /dev/sata/home
/dev/sata/home: symbolic link to `../dm-3'
merkaba:~> file -sk /dev/dm-3
/dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384, leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a, 132303151104/343597383680 bytes used, 2 devices

Quoted 1 line

Ciao,

I have seen that behavior as well. It turned out that the directory was actually a subvolume. Are you sure it is a directory?

--
Torbjørn

Martin Steigerwald

author

2014-07-15 09:43 UTC #2 lore raw

Am Dienstag, 15. Juli 2014, 11:25:15 schrieb Torbjørn:

Quoted 55 lines · Martin Steigerwald

Quoted 54 lines
Hello!
This is with 3.16-rc4 – stepped back to this one after having two hangs in one day with 3.16-rc5, see other thread started by me:
martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin 14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1>
Whats this?
I had this weeks ago already and just moved it out of the way at that time,
just now stumbled upon it again.
This is the same BTRFS Dual SSD RAID as mentioned in the other post:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
         Total devices 2 FS bytes used 123.20GiB
         devid    1 size 160.00GiB used 159.98GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 159.98GiB path /dev/dm-3
Btrfs v3.14.1
merkaba:~#1> btrfs fi df /home
Data, RAID1: total=154.95GiB, used=120.61GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=2.59GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> df -hT /home
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/msata-home btrfs 320G 247G 69G 79% /home
merkaba:~> file -sk /dev/sata/home
/dev/sata/home: symbolic link to `../dm-3'
merkaba:~> file -sk /dev/dm-3
/dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384, leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a, 132303151104/343597383680 bytes used, 2 devices

[…]

Quoted 3 lines

I have seen that behavior as well. It turned out that the directory was actually a subvolume.
Are you sure it is a directory?

Currently I have only one subvolume called /home (to make snapshots of /home outside of /home)

merkaba:~> btrfs subvol list /home
ID 256 gen 147145 top level 5 path home

Thanks,

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Hugo Mills

2014-07-15 10:01 UTC #3 lore raw

In reply to Martin Steigerwald

Quoted 20 lines · Martin Steigerwald

Hello!
This is with 3.16-rc4 – stepped back to this one after having two hangs in one day with 3.16-rc5, see other thread started by me:
martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin 14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1>

Quoted 4 lines

Whats this?
I had this weeks ago already and just moved it out of the way at that time, just now stumbled upon it again.

That is symptomatic of a bug from a couple of kernel versions ago
(now fixed, so it won't happen again). It it is that bug, then btrfs check will report something along the lines of "directory isize wrong", and the problem can be fixed by running a btrfs check --repair. If you get anything else from btrfs check (or it checks cleanly), then let us know first.

Hugo.

Signature · 3 lines

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- If you see something, say nothing and drink to forget ---

Martin Steigerwald

author

2014-08-27 10:00 UTC #4 lore raw

Am Dienstag, 15. Juli 2014, 11:01:25 schrieb Hugo Mills:

Quoted 27 lines · Martin Steigerwald

Quoted 20 lines
Hello!
This is with 3.16-rc4 – stepped back to this one after having two hangs in one day with 3.16-rc5, see other thread started by me:
martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin 14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty martin@merkaba:~/Zeit/undeletable/db_data#1>
Quoted 5 lines
Whats this?
I had this weeks ago already and just moved it out of the way at that time,
just now stumbled upon it again.

Quoted 6 lines

That is symptomatic of a bug from a couple of kernel versions ago
(now fixed, so it won't happen again). It it is that bug, then btrfs check will report something along the lines of "directory isize wrong", and the problem can be fixed by running a btrfs check --repair. If you get anything else from btrfs check (or it checks cleanly), then let us know first.

Thanks, Hugo. Just a heads up I was able to fix the issue and how btrfs check improved from 3.14.1 to 3.16 :)

Some time ago I can btrfs check 3.14.1 and while it fixed this undeleteable file it was able to fix another issue with that BTRFS fs:

enabling repair mode
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated checking fs roots
reset isize for dir 435395 root 256
reset isize for dir 435397 root 256
reset isize for dir 450598 root 256
reset isize for dir 2654270 root 256
btrfs: volumes.c:948: btrfs_alloc_chunk: Assertion `!(ret)' failed.
zsh: abort btrfs check --repair /dev/msata/home

This one was still there:

Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated checking fs roots
root 256 inode 5759476 errors 200, dir isize wrong
found 37709498062 bytes used err is 1
total csum bytes: 130574180
total tree bytes: 2897543168
total fs tree bytes: 2396323840
total extent tree bytes: 309051392
btree space waste bytes: 538971257
file data blocks allocated: 2757585866752
referenced 135559811072
Btrfs v3.14.1

And it didn´t want to fix it at all, asserting all along.

However

merkaba:/home/martin/Zeit> rm -r undeletable/

worked okay.

And now btrfs check 3.16 also cleaned up the last error:

merkaba:~> btrfs check /dev/msata/home Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
checking fs roots
root 256 inode 5759476 errors 200, dir isize wrong found 51387055905 bytes used err is 1 total csum bytes: 128640084
total tree bytes: 2901934080
total fs tree bytes: 2398830592
total extent tree bytes: 306806784
btree space waste bytes: 547937419
file data blocks allocated: 2744207491072
referenced 129598066688
Btrfs v3.16

merkaba:~#1> btrfs check --repair /dev/msata/home
enabling repair mode
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated checking fs roots
reset isize for dir 5759476 root 256
checking csums
checking root refs
found 51387055905 bytes used err is 0
total csum bytes: 128640084
total tree bytes: 2901934080
total fs tree bytes: 2398830592

I made sure the error is gone by doing another check and it is gone indeed.

BTW I find "found 51387055905 bytes used err is 0" a bit ambigious. While one can say "err is 0" means no error, people may wonder whether there is an error or not. I assume that this is display a mismatch between the found bytes and the bytes that are *supposed* to be there.

This all was on 3.17-rc2 with fixcompwrite patch v3 from Liu.

Thanks,

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7