Strange "rm -fr" issue

DISCUSSION Markus Trippelsdorf 3 participants 5 messages Dec 29, 2013 – Dec 30, 2013

Share & tools

Markus Trippelsdorf

author

2013-12-29 14:30 UTC #0 lore raw

I came across the following strange issue today:

markus@x4 tmp % rm -fr boost
rm: cannot remove ‘boost/results/boost/bin.v2/libs’: Directory not empty markus@x4 tmp % ll boost/results/boost/bin.v2/libs
total 68K
drwxr-xr-x 17 markus markus 4.0K Dec 29 15:01 .

drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
total 140K
drwxr-xr-x 35 markus markus 4.0K Dec 29 15:01 .
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 config
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 predef
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 preprocessor
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 program_options
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_map
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_tree
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 proto
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ptr_container
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 random
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 range
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ratio
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 rational
drwxr-xr-x  5 markus markus 4.0K Dec 29 15:01 regex
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 scope_exit
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 signals
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 signals2
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 smart_ptr
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 spirit
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test

markus@x4 tmp % rm -fr boost
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
ls: cannot access boost/results/boost/bin.v2/libs: No such file or directory markus@x4 tmp %

fsck.ext4 shows nothing. Kernel is latest git. coreutils-8.22.
/dev/sda ext4 1.8T 743G 1.1T 41% /var
/dev/sda on /var type ext4 (rw,relatime,data=ordered)

Signature · 5 lines

Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Andreas Dilger

2013-12-29 18:47 UTC #1 lore raw

If you are able to reproduce this issue (extract the same tree and then
try "rm -fr" again) please run this under strace to see how many entries are returned during readdir from the kernel.

It could be a bug in the ext4 readdir code or it could be a bug in glibc or coreutils.

Cheers, Andreas

Quoted 62 lines · Markus Trippelsdorf

I came across the following strange issue today:

drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
total 140K
drwxr-xr-x 35 markus markus 4.0K Dec 29 15:01 .
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 config
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 predef
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 preprocessor
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 program_options
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_map
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_tree
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 proto
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ptr_container
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 random
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 range
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ratio
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 rational
drwxr-xr-x  5 markus markus 4.0K Dec 29 15:01 regex
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 scope_exit
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 signals
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 signals2
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 smart_ptr
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 spirit
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test

Quoted 15 lines

markus@x4 tmp % rm -fr boost
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
ls: cannot access boost/results/boost/bin.v2/libs: No such file or directory markus@x4 tmp %
fsck.ext4 shows nothing. Kernel is latest git. coreutils-8.22.
/dev/sda ext4 1.8T 743G 1.1T 41% /var
/dev/sda on /var type ext4 (rw,relatime,data=ordered)
Signature · 5 lines
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf

author

2013-12-30 08:10 UTC #2 lore raw

Quoted 6 lines · Andreas Dilger

If you are able to reproduce this issue (extract the same tree and then try "rm -fr" again) please run this under strace to see how many entries are returned during readdir from the kernel.
It could be a bug in the ext4 readdir code or it could be a bug in glibc or coreutils.

Unfortunately the issue is not reproducible and therefore impossible to debug. I just thought to post it nevertheless, just to see if perhaps someone else encounters the same problem.

Signature · 1 line

Markus

Jan Kara

2013-12-30 22:17 UTC #3 lore raw

In reply to Markus Trippelsdorf

Quoted 60 lines · Markus Trippelsdorf

I came across the following strange issue today:

drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
total 140K
drwxr-xr-x 35 markus markus 4.0K Dec 29 15:01 .
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 config
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 predef
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 preprocessor
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 program_options
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_map
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_tree
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 proto
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ptr_container
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 random
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 range
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ratio
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 rational
drwxr-xr-x  5 markus markus 4.0K Dec 29 15:01 regex
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 scope_exit
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 signals
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 signals2
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 smart_ptr
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 spirit
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test

Quoted 4 lines

markus@x4 tmp % rm -fr boost
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
ls: cannot access boost/results/boost/bin.v2/libs: No such file or directory markus@x4 tmp %

Hum, my guess would be that some other process has been adding files to
the directory while the first 'rm -fr' was running. Thus rm got a list of files in 'libs' to delete, deleted them, tried to delete 'libs' and failed because the other process has added some new files / dirs in the mean time. By the time you run second rm, the creating process has apparently finished.

Honza

Signature · 6 lines

Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf

author

2013-12-30 22:22 UTC #4 lore raw

Quoted 65 lines · Jan Kara

Quoted 60 lines · Markus Trippelsdorf

I came across the following strange issue today:

drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
total 140K
drwxr-xr-x 35 markus markus 4.0K Dec 29 15:01 .
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 ..
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 config
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 move
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 mpl
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 msm
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_array
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 multi_index
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:00 multiprecision
drwxr-xr-x  6 markus markus 4.0K Dec 29 15:01 numeric
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 optional
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 parameter
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 phoenix
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 polygon
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 pool
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 predef
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 preprocessor
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 program_options
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_map
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 property_tree
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 proto
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ptr_container
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 python
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 random
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 range
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 ratio
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 rational
drwxr-xr-x  5 markus markus 4.0K Dec 29 15:01 regex
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 scope_exit
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 serialization
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 signals
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 signals2
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 smart_ptr
drwxr-xr-x  4 markus markus 4.0K Dec 29 15:01 spirit
drwxr-xr-x  3 markus markus 4.0K Dec 29 15:01 test

Quoted 4 lines

markus@x4 tmp % rm -fr boost
markus@x4 tmp % ll boost/results/boost/bin.v2/libs
ls: cannot access boost/results/boost/bin.v2/libs: No such file or directory markus@x4 tmp %

Quoted 6 lines

Hum, my guess would be that some other process has been adding files to
the directory while the first 'rm -fr' was running. Thus rm got a list of files in 'libs' to delete, deleted them, tried to delete 'libs' and failed because the other process has added some new files / dirs in the mean time. By the time you run second rm, the creating process has apparently finished.

I'm pretty sure that this wasn't the case. The boost test suite had finished and the machine was idle for a while, before I tried to remove the directory...

Signature · 5 lines

Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html