soft lockup - CPU#X stuck for 23s! (vfs, autofs, vserver)

2.6.38-3.x Paweł Sikora 4 participants 9 messages Sep 23, 2012 – Nov 15, 2012 VFS

Share & tools

Paweł Sikora

author

2012-09-23 06:18 UTC #0 lore raw

Hi,

a long time ago i reported an ugly soft lock of heavy loaded opteron system without usable backtraces :( recently, i've logged on 3.4.6 via serial console backtraces from all 16 cores which show some kind of vfs lock (http://pluto.agmk.net/kernel/oops.txt). this lock occurs on heavy loaded system with frequent autofs unmount (timeout=1s) in action with vserver patch applied.

i've isolated the vserver function that exposes problem: mnt_is_reachable()
http://vserver.13thfloor.at/Experimental/patch-3.4.6-vs2.3.3.6.diff:

static int mnt_is_reachable(struct vfsmount *vfsmnt)

{
	struct path root;
	struct dentry *point;
	struct mount *mnt = real_mount(vfsmnt);
	struct mount *root_mnt;
	int ret;

	if (mnt == mnt->mnt_ns->root)
		return 1;

	br_read_lock(vfsmount_lock);
	root = current->fs->root;
	root_mnt = real_mount(root.mnt);
	point = root.dentry;

	while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
		point = mnt->mnt_mountpoint;
		mnt = mnt->mnt_parent;
	}

	ret = (mnt == root_mnt) && is_subdir(point, root.dentry);

	br_read_unlock(vfsmount_lock);

	return ret;
}

the vserver developer (Herbert Poetzl) said that locking scheme used in this function looks correct, so it might be a hidden vfs bug accidentaly exposed by vserver patch. i'm not a kernel developer, so i'd like to ask you for help in solving this problem. afaics the problem starts with 2.6.38 release which introduced some major vfs changes.

please CC me on reply.

BR,
Paweł.

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Linus Torvalds

2012-09-24 01:10 UTC #1 lore raw

Quoted 2 lines · Paweł Sikora

        br_read_lock(vfsmount_lock);

The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.

And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.

Why do you have that 1s timeout? Insane.

Linus

Paweł Sikora

author

2012-09-24 05:23 UTC #2 lore raw

Quoted 8 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:

Quoted 2 lines · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.

Quoted 5 lines

And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.

1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup). can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?

BR,
Paweł.

ps).
i'm adding Herbert to CC.

Herbert Poetzl

2012-09-24 11:23 UTC #3 lore raw

Quoted 16 lines · Paweł Sikora

Quoted 9 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:
Quoted 1 line · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.
Quoted 5 lines
And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.

Quoted 8 lines

1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup).
can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?

currently we do:

        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);

and we have been considering to move the br_read_unlock() right before the is_subdir() call

if there are any suggestions how to achieve the same with less locking I'm all ears ...

best,
Herbert

Quoted 5 lines

BR,
Paweł.
ps).
i'm adding Herbert to CC.

Linus Torvalds

2012-09-24 17:22 UTC #4 lore raw

Quoted 16 lines · Herbert Poetzl

currently we do:

        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);

Quoted 2 lines

and we have been considering to move the br_read_unlock() right before the is_subdir() call

So the read lock itself should not cause problems. We have tons of high-frequency read-lockers, over quite big areas.

But exactly because the readlockers are so high-frequency, I'd expect any problems to be *found* by read-lockers.

But the *cause* would likely be
- missing unlocks
- write-locks
- recursion on the locks (including read-locks: they are actually
per-cpu spinlocks)

but I'd have expected lockdep to find anything obvious like that.

If the locking itself is fine, maybe the loop above (or the is_subdir()) is infinite due to mnt->mnt_parent somehow becoming a circular loop. Maybe due to corrupt data structures traversed inside the lock, causing infinite loops..

I really don't know the vserver patches, it would be much more interesting if you can recreate the problems using a standard kernel.

Linus

Eric W. Biederman

2012-09-24 18:17 UTC #5 lore raw

In reply to Herbert Poetzl

Quoted 41 lines · Herbert Poetzl

Quoted 16 lines · Paweł Sikora
Quoted 9 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:
Quoted 1 line · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.
Quoted 5 lines
And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.
Quoted 8 lines
1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup).
can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?
currently we do:
        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);

Quoted 5 lines

and we have been considering to move the br_read_unlock() right before the is_subdir() call
if there are any suggestions how to achieve the same with less locking I'm all ears ...

Herbert, why do you need to filter the mounts that show up in a mount namespace at all? I would think a far more performant and simpler solution would be to just use mount namespaces without unwanted mounts.

I'd like to blame this on the silly rcu_barrier in deactivate_locked_super that should really be in the module remove path, but that happens after we drop the br_write_lock.

The kernel take br_read_lock(&vfs_mount_lokck) during every rcu path lookup so mnt_is_reachable isn't particular crazy just for taking the lock.

I am with Linus on this one. Paweł even 60s for your mount timeout looks too short for your workload. All of the readers that take br_read_lock(&vfsmount_lock) seem to be showing up in your oops. The only thing that seems to make sense is you have a lot of unmount activity running back to back, keeping the lock write held.

The only other possible culprit I can see is that it looks like mnt_is_reachable changes reading /proc/mounts to be something worse than linear in the number of mounts and reading /proc/mounts starts taking the vfsmount_lock. All minor things but when you are pushing things hard they look like things that would add up.

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Herbert Poetzl

2012-09-25 05:05 UTC #6 lore raw

Quoted 43 lines · Eric W. Biederman

Quoted 40 lines · Herbert Poetzl
Quoted 16 lines · Paweł Sikora
Quoted 9 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:
Quoted 1 line · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.
Quoted 5 lines
And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.
Quoted 8 lines
1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup).
can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?
currently we do:
        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);

Quoted 8 lines

Quoted 5 lines
and we have been considering to move the br_read_unlock() right before the is_subdir() call
if there are any suggestions how to achieve the same with less locking I'm all ears ...
Herbert, why do you need to filter the mounts that show up in a mount namespace at all?

that is actually a really good question!

Quoted 2 lines

I would think a far more performant and simpler solution would be to just use mount namespaces without unwanted mounts.

we had this mechanism for many years, long before the mount namespaces existed, and I vaguely remember that early versions didn't get the proc entries right either

I took a quick look at the code and I think we can drop the mnt_is_reachable() check and/or make it conditional on setups without a mount namespace in place in the near future (thanks for the input, really appreciated!)

Quoted 22 lines

I'd like to blame this on the silly rcu_barrier in deactivate_locked_super that should really be in the module remove path, but that happens after we drop the br_write_lock.
The kernel take br_read_lock(&vfs_mount_lokck) during every rcu path lookup so mnt_is_reachable isn't particular crazy just for taking the lock.
I am with Linus on this one. Paweł even 60s for your mount timeout looks too short for your workload. All of the readers that take br_read_lock(&vfsmount_lock) seem to be showing up in your oops. The only thing that seems to make sense is you have a lot of unmount activity running back to back, keeping the lock write held.
The only other possible culprit I can see is that it looks like mnt_is_reachable changes reading /proc/mounts to be something worse than linear in the number of mounts and reading /proc/mounts starts taking the vfsmount_lock. All minor things but when you are pushing things hard they look like things that would add up.
Eric

Paweł Sikora

author

2012-11-15 18:48 UTC #7 lore raw

Quoted 58 lines · On Tuesday 25 of September 2012 07:05:59 Herbert Poetzl wrote:

Quoted 43 lines · Eric W. Biederman
Quoted 40 lines · Herbert Poetzl
Quoted 16 lines · Paweł Sikora
Quoted 9 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:
Quoted 1 line · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.
Quoted 5 lines
And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.
Quoted 8 lines
1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup).
can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?
currently we do:
        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);
Quoted 8 lines
Quoted 5 lines
and we have been considering to move the br_read_unlock() right before the is_subdir() call
if there are any suggestions how to achieve the same with less locking I'm all ears ...
Herbert, why do you need to filter the mounts that show up in a mount namespace at all?
that is actually a really good question!
Quoted 2 lines
I would think a far more performant and simpler solution would be to just use mount namespaces without unwanted mounts.

Quoted 8 lines

we had this mechanism for many years, long before the mount namespaces existed, and I vaguely remember that early versions didn't get the proc entries right either
I took a quick look at the code and I think we can drop the mnt_is_reachable() check and/or make it conditional on setups without a mount namespace in place in the near future (thanks for the input, really appreciated!)

Hi,

Herbert, can i just drop this mnt_is_reachable() method from vserver patch? this issue hasn't been solved for several months now. i can live without this problematic security-through-obscurity feature on my heavy loaded machines. .

Quoted 22 lines

Quoted 22 lines
I'd like to blame this on the silly rcu_barrier in deactivate_locked_super that should really be in the module remove path, but that happens after we drop the br_write_lock.
The kernel take br_read_lock(&vfs_mount_lokck) during every rcu path lookup so mnt_is_reachable isn't particular crazy just for taking the lock.
I am with Linus on this one. Paweł even 60s for your mount timeout looks too short for your workload. All of the readers that take br_read_lock(&vfsmount_lock) seem to be showing up in your oops. The only thing that seems to make sense is you have a lot of unmount activity running back to back, keeping the lock write held.
The only other possible culprit I can see is that it looks like mnt_is_reachable changes reading /proc/mounts to be something worse than linear in the number of mounts and reading /proc/mounts starts taking the vfsmount_lock. All minor things but when you are pushing things hard they look like things that would add up.
Eric

Herbert Poetzl

2012-11-15 19:40 UTC #8 lore raw

Quoted 68 lines · Paweł Sikora

Quoted 58 lines · On Tuesday 25 of September 2012 07:05:59 Herbert Poetzl wrote:
Quoted 43 lines · Eric W. Biederman
Quoted 40 lines · Herbert Poetzl
Quoted 16 lines · Paweł Sikora
Quoted 9 lines · On Sunday 23 of September 2012 18:10:30 Linus Torvalds wrote:
Quoted 1 line · Paweł Sikora
        br_read_lock(vfsmount_lock);
The vfsmount_lock is a "local-global" lock, where a read-lock is rather cheap and takes just a per-cpu lock, but the downside is that a write-lock is *very* expensive, and can cause serious trouble.
Quoted 5 lines
And the write lock is taken by the [un]mount() paths. Do *not* do crazy things. If you do some insane "unmount and remount autofs" on a 1s granularity, you're doing insane things.
Why do you have that 1s timeout? Insane.
Quoted 8 lines
1s unmount timeout is *only* for fast bug reproduction (in few seconds after opteron startup) and testing potential patches. normally with 60s timeout it happens in few minutes..hours (depends on machine i/o+cpu load) and makes server unusable (permament soft-lockup).
can we redesign vserver's mnt_is_reachable() for better locking to avoid total soft-lockup?
currently we do:
        br_read_lock(&vfsmount_lock);
        root = current->fs->root;
        root_mnt = real_mount(root.mnt);
        point = root.dentry;

        while ((mnt != mnt->mnt_parent) && (mnt != root_mnt)) {
                point = mnt->mnt_mountpoint;
                mnt = mnt->mnt_parent;
        }

        ret = (mnt == root_mnt) && is_subdir(point, root.dentry);
        br_read_unlock(&vfsmount_lock);
Quoted 8 lines
Quoted 5 lines
and we have been considering to move the br_read_unlock() right before the is_subdir() call
if there are any suggestions how to achieve the same with less locking I'm all ears ...
Herbert, why do you need to filter the mounts that show up in a mount namespace at all?
that is actually a really good question!
Quoted 2 lines
I would think a far more performant and simpler solution would be to just use mount namespaces without unwanted mounts.
Quoted 8 lines
we had this mechanism for many years, long before the mount namespaces existed, and I vaguely remember that early versions didn't get the proc entries right either
I took a quick look at the code and I think we can drop the mnt_is_reachable() check and/or make it conditional on setups without a mount namespace in place in the near future (thanks for the input, really appreciated!)

Quoted 7 lines

Hi,
Herbert, can i just drop this mnt_is_reachable() method from vserver patch? this issue hasn't been solved for several months now. i can live without this problematic security-through-obscurity feature on my heavy loaded machines.

sure, if you are aware of the implications, you can simply remove the check ...

best,
Herbert

Quoted 15 lines

Quoted 14 lines
Quoted 14 lines
I'd like to blame this on the silly rcu_barrier in deactivate_locked_super that should really be in the module remove path, but that happens after we drop the br_write_lock.
The kernel take br_read_lock(&vfs_mount_lokck) during every rcu path lookup so mnt_is_reachable isn't particular crazy just for taking the lock.
I am with Linus on this one. Paweł even 60s for your mount timeout looks too short for your workload. All of the readers that take br_read_lock(&vfsmount_lock) seem to be showing up in your oops. The only thing that seems to make sense is you have a lot of unmount activity running back to back, keeping the lock write held.

Quoted 7 lines

Quoted 7 lines
Quoted 7 lines
The only other possible culprit I can see is that it looks like mnt_is_reachable changes reading /proc/mounts to be something worse than linear in the number of mounts and reading /proc/mounts starts taking the vfsmount_lock. All minor things but when you are pushing things hard they look like things that would add up.
Eric

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html