Skip to contentSkip to conversation

lockdep splat after mounting overlayfs over overlayfs

overlayfs Konstantin Khlebnikov 3 participants 4 messages Jun 24, 2015 – Jun 26, 2015

lore
Share & tools
lore

Konstantin Khlebnikov

author
#0 lore raw

I've accidentally mounted one overlayfs over another and got obvious warning from lockdep: i_mutex lockdep classes are per-fs-type.

# mount -t overlay overlay 1 -o upperdir=1_upper,workdir=1_work,lowerdir=1_lower
# mount -t overlay overlay 2 -o upperdir=2_upper,workdir=2_work,lowerdir=1
# ls 2
[  305.098705] =============================================
[  305.099011] [ INFO: possible recursive locking detected ]
[  305.099011] 4.1.0+ #76 Not tainted
[  305.099011] ---------------------------------------------
[  305.099011] ls/3070 is trying to acquire lock:
[  305.099011]  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff811b9ac0>] iterate_dir+0x70/0x130
[  305.099011]
[  305.099011] but task is already holding lock:
[  305.099011]  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff811b9ac0>] iterate_dir+0x70/0x130
[  305.099011]
[  305.099011] other info that might help us debug this:
[  305.099011]  Possible unsafe locking scenario:
[  305.099011]
[  305.099011]        CPU0
[  305.099011]        ----
[  305.099011]   lock(&sb->s_type->i_mutex_key#13);
[  305.099011]   lock(&sb->s_type->i_mutex_key#13);
[  305.099011]
[  305.099011]  *** DEADLOCK ***
[  305.099011]
[  305.099011]  May be due to missing lock nesting notation
[  305.099011]
[  305.099011] 1 lock held by ls/3070:
[  305.099011]  #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff811b9ac0>] iterate_dir+0x70/0x130
[  305.099011]
[  305.099011] stack backtrace:
[  305.099011] CPU: 2 PID: 3070 Comm: ls Not tainted 4.1.0+ #76
[  305.099011] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[  305.099011]  ffff880037f40000 ffff880037fa3b48 ffffffff81991179 0000000000000007
[  305.099011]  ffffffff825ac1a0 ffff880037fa3c38 ffffffff810a92ea 0000000000000000
[  305.099011]  0000000000000000 ffff8802110eaea0 ffff88021118c720 ffff880214576710
[  305.099011] Call Trace:
[  305.099011]  [<ffffffff81991179>] dump_stack+0x4c/0x65
[  305.099011]  [<ffffffff810a92ea>] __lock_acquire+0x91a/0x1f60
[  305.099011]  [<ffffffff811d0a1b>] ? inode_to_bdi+0x2b/0x60
[  305.099011]  [<ffffffff810ab0d1>] lock_acquire+0xd1/0x290
[  305.099011]  [<ffffffff811b9ac0>] ? iterate_dir+0x70/0x130
[  305.099011]  [<ffffffff8199b2c8>] mutex_lock_killable_nested+0x68/0x440
[  305.099011]  [<ffffffff811b9ac0>] ? iterate_dir+0x70/0x130
[  305.099011]  [<ffffffff811b9ac0>] ? iterate_dir+0x70/0x130
[  305.099011]  [<ffffffff811b9ac0>] iterate_dir+0x70/0x130
[  305.099011]  [<ffffffff812fcf05>] ovl_dir_read_merged+0x125/0x160
[  305.099011]  [<ffffffff812fcc10>] ? ovl_cache_entry_new+0x110/0x110
[  305.099011]  [<ffffffff812fd45d>] ovl_iterate+0x14d/0x1e0
[  305.099011]  [<ffffffff811b9afc>] iterate_dir+0xac/0x130
[  305.099011]  [<ffffffff811c4d64>] ? __fget_light+0x74/0xa0
[  305.099011]  [<ffffffff811b9c87>] SyS_getdents+0x87/0x100
[  305.099011]  [<ffffffff811b9920>] ? filldir64+0x140/0x140
[  305.099011]  [<ffffffff8199e5ae>] entry_SYSCALL_64_fastpath+0x12/0x76
Signature · 1 line
Konstantin
#1 lore raw
Quoted 7 lines

I've accidentally mounted one overlayfs over another and got obvious warning from lockdep: i_mutex lockdep classes are per-fs-type.

# mount -t overlay overlay 1 -o
upperdir=1_upper,workdir=1_work,lowerdir=1_lower
# mount -t overlay overlay 2 -o upperdir=2_upper,workdir=2_work,lowerdir=1
# ls 2

This reporting is positive, we are not in deadlock situation actually. The "2" dir of overlayfs will call iterate_dir->"1" dir of overlayfs->iterate_dir, and the nest iterate_dir happened on the same file system, so the warning came out.

We'd better make the lower and upper in different fs instance, and this warning will disappear.

And this lockdep warning happened when the nest iterate_dir call of same fs(I mean the same super block). The function check_deadlock in lockdep.c will report the nest lock of same process. If we make the 2_upper and 2_work in a different fs, no warning comes out.

Thanks,

Signature · 8 lines
George Wang 王旭

Kernel Quantity Engineer
Red Hat Software (Beijing) Co.,Ltd
IRC:xuw
Tel:+86-010-62608041
Phone:15901231579
9/F, Tower C, Raycom
#2 lore raw
Quoted 12 lines · Xu Wang
Quoted 7 lines

I've accidentally mounted one overlayfs over another and got obvious warning from lockdep: i_mutex lockdep classes are per-fs-type.

# mount -t overlay overlay 1 -o
upperdir=1_upper,workdir=1_work,lowerdir=1_lower
# mount -t overlay overlay 2 -o upperdir=2_upper,workdir=2_work,lowerdir=1
# ls 2

This reporting is positive, we are not in deadlock situation actually. The "2" dir of overlayfs will call iterate_dir->"1" dir of overlayfs->iterate_dir, and the nest iterate_dir happened on the same file system, so the warning came out.

Quoted 7 lines

We'd better make the lower and upper in different fs instance, and this warning will disappear.

And this lockdep warning happened when the nest iterate_dir call of same fs(I mean the same super block). The function check_deadlock in lockdep.c will report the nest lock of same process. If we make the 2_upper and 2_work in a different fs, no warning comes out.

Yep, it's not a deadlock. As I mentioned lockdep classes are per-fs-type so lockdep cannot see difference between i_mutexes on different sb of the same type.
But anyway this looks messy.

Probably it's safer to forbid overlayfs as lower or upper mount for overlayfs because this have no sense. Nesting anyway is limited by the depth of kernel stack and sb->s_stack_depth.

Or overlayfs could detect this situation and substitute layers of underlying overlayfs into its own lower layers in appropriate place.

Quoted 17 lines

Thanks,

--
George Wang 王旭

Kernel Quantity Engineer
Red Hat Software (Beijing) Co.,Ltd
IRC:xuw
Tel:+86-010-62608041
Phone:15901231579
9/F, Tower C, Raycom
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 25 lines · Xu Wang
Quoted 18 lines
Quoted 7 lines

I've accidentally mounted one overlayfs over another and got obvious warning from lockdep: i_mutex lockdep classes are per-fs-type.

# mount -t overlay overlay 1 -o
upperdir=1_upper,workdir=1_work,lowerdir=1_lower
# mount -t overlay overlay 2 -o upperdir=2_upper,workdir=2_work,lowerdir=1
# ls 2

This reporting is positive, we are not in deadlock situation actually. The "2" dir
of overlayfs will call iterate_dir->"1" dir of overlayfs->iterate_dir, and the nest
iterate_dir happened on the same file system, so the warning came out.

We'd better make the lower and upper in different fs instance, and this warning
will disappear.

Quoted 5 lines

And this lockdep warning happened when the nest iterate_dir call of same fs(I
mean the same super block). The function check_deadlock in lockdep.c will report the nest lock of same process. If we make the 2_upper and 2_work in a different fs, no warning comes out.

Quoted 5 lines

Yep, it's not a deadlock. As I mentioned lockdep classes are per-fs-type so lockdep cannot see difference between i_mutexes on different sb of the same type.
But anyway this looks messy.

yes, you are right. The i_mutex_class is file_system_tye scale. I was puzzled by the debug_locks mechanism during my quick tests. The nest iterate_dir is overlay dir, neither upper nor lower.

Quoted 7 lines

Probably it's safer to forbid overlayfs as lower or upper mount for overlayfs because this have no sense. Nesting anyway is limited by the depth of kernel stack and sb->s_stack_depth.

Or overlayfs could detect this situation and substitute layers of underlying overlayfs into its own lower layers in appropriate place.

Can we add the lockdep_off/lockdep_on in this situation? For we know this is just the false positive reporting of lockdep.

Thanks,

Xu Wang