Skip to contentSkip to conversation

About the try to remove cross-release feature entirely by Ingo

PATCH In review Byungchul Park 10 participants 40 messages Dec 13, 2017 – Jan 5, 2018

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we
might get false positives.

For (1), we don't have to classify locks 100%
properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

In addition, I want to say that the current level of classification is much less than 100% but, since we have annotated well to suppress wrong reports by rough classifications, finally it does not come into view by original lockdep for now.

But since cross-release makes the dependency graph more fine-grained, it easily comes into view.

Even w/o cross-release, it can happen by adding additional dependencies connecting two roughly classified lock classes in the future.

Furthermore, I can see many places in kernel to consider wait_for_completion() using manual lock_acquire()/lock_release() and the rate using it raises.

In other words, even without cross-release, the more we add manual annotations for wait_for_completion() the more we definitely suffer same problems someday, we are facing now through cross-release.

Therefore, I want to say the fundamental problem comes from classification, not cross-release specific. Of course, since cross-release accelerates the condition, I agree with it to be off for now.

Signature · 2 lines
Thanks,
Byungchul
#2 lore raw

T24gV2VkLCAyMDE3LTEyLTEzIGF0IDE2OjEzICswOTAwLCBCeXVuZ2NodWwgUGFyayB3cm90ZToN Cj4gSW4gYWRkaXRpb24sIEkgd2FudCB0byBzYXkgdGhhdCB0aGUgY3VycmVudCBsZXZlbCBvZg0K PiBjbGFzc2lmaWNhdGlvbiBpcyBtdWNoIGxlc3MgdGhhbiAxMDAlIGJ1dCwgc2luY2Ugd2UNCj4g aGF2ZSBhbm5vdGF0ZWQgd2VsbCB0byBzdXBwcmVzcyB3cm9uZyByZXBvcnRzIGJ5DQo+IHJvdWdo IGNsYXNzaWZpY2F0aW9ucywgZmluYWxseSBpdCBkb2VzIG5vdCBjb21lIGludG8NCj4gdmlldyBi eSBvcmlnaW5hbCBsb2NrZGVwIGZvciBub3cuDQoNClRoZSBMaW51eCBrZXJuZWwgaXMgbm90IGEg dmVoaWNsZSBmb3IgZXhwZXJpbWVudHMuIFRoZSBtYWpvcml0eSBvZiBmYWxzZQ0KcG9zaXRpdmVz IHNob3VsZCBoYXZlIGJlZW4gZml4ZWQgYmVmb3JlIHRoZSBjcm9zc3JlbGVhc2UgcGF0Y2hlcyB3 ZXJlIHNlbnQNCnRvIExpbnVzLg0KDQpCYXJ0Lg==

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
In reply to Byungchul Park
Quoted 4 lines · Byungchul Park

Therefore, I want to say the fundamental problem
comes from classification, not cross-release
specific.

You keep saying that it is "just" a matter of classificaion.

However, it is not obvious how to do the classification in a sane manner. And this is why I keep pointing out that there is no documentation on how to do this, and somehow you never respond to this point....

In the case where you have multiple unrelated subsystems that can be stacked in different ways, with potentially multiple instances stacked on top of each other, it is not at all clear to me how this problem should be solved.

It was said on one of these threads (perhaps by you, perhaps by someone else), that we can't expect the lockdep maintainers to understand all of the subsystems in the kernels, and so therefore it must be up to the subsystem maintainers to classify the locks. I interpreted this as the lockdep maintainers saying, "hey, not my fault, it's the subsystem maintainer's fault for not properly classifying the locks" --- and thus dumping the responsibility in the subsystem maintainers' laps.

I don't know if the situation is just that lockdep is insufficiently documented, and with the proper tutorial, it would be obvious how to solve the classification problem.

Or, if perhaps, there *is* no way to solve the classification problem, at least not in a general form.

For example --- suppose we have a network block device on which there is an btrfs file system, which is then exported via NFS. Now all of the TCP locks will be used twice for two different instances, once for the TCP connection for the network block device, and then for the NFS export.

How exactly are we supposed to classify the locks to make it all work?

Or the loop device built on top of an ext4 file system which on a LVM/device mapper device. And suppose the loop device is then layered with a dm-error device for regression testing, and with another ext4 file system on top of that?

How exactly are we supposed to classify the locks in that situation? Where's the documentation and tutorials which explain how to make this work, if the responsibility is going to be dumped on the subsystem maintainers' laps? Or if the lockdep maintainers are expected to fix and classify all of these locks, are you volunteering to do this?

How hard is it exactly to do all of this classification work, no matter whose responsibility it will ultimately be?

And if the answer is that it is too hard, then let me gently suggest to you that perhaps, if this is a case, that maybe this is a fundamental and fatal flaw with the cross-release and completion lockdep feature?

Best regards,

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 7 lines · Theodore Ts'o
Quoted 4 lines · Byungchul Park

Therefore, I want to say the fundamental problem
comes from classification, not cross-release
specific.

You keep saying that it is "just" a matter of classificaion.

But, it's a fact.

Quoted 4 lines

However, it is not obvious how to do the classification in a sane manner. And this is why I keep pointing out that there is no documentation on how to do this, and somehow you never respond to this point....

I can write a document explaining what lock class is but.. I cannot explain how to assign it perfectly since there's no right answer. It's something we need to improve more and more.

Quoted 4 lines

In the case where you have multiple unrelated subsystems that can be stacked in different ways, with potentially multiple instances stacked on top of each other, it is not at all clear to me how this problem should be solved.

I cannot give you a perfect solution immediately. I know, and as you know, it's a very difficult problem to solve.

Quoted 8 lines

It was said on one of these threads (perhaps by you, perhaps by someone else), that we can't expect the lockdep maintainers to understand all of the subsystems in the kernels, and so therefore it must be up to the subsystem maintainers to classify the locks. I interpreted this as the lockdep maintainers saying, "hey, not my fault, it's the subsystem maintainer's fault for not properly classifying the locks" --- and thus dumping the responsibility in the subsystem maintainers' laps.

Sorry to say, making you feel like that.

Precisely speaking, the responsibility for something caused by cross-release is on me, and the responsibility for something caused by lockdep itselt is on lockdep.

I meant, in the current way to assign lock class automatically, it's inevitable for someone to annotate places manually, and it can be done best by each expert. But, anyway fundamentally I think the responsibility is on lockdep.

Quoted 6 lines

I don't know if the situation is just that lockdep is insufficiently documented, and with the proper tutorial, it would be obvious how to solve the classification problem.

Or, if perhaps, there *is* no way to solve the classification problem, at least not in a general form.

Agree. It's a very difficult one to solve.

Quoted 6 lines

For example --- suppose we have a network block device on which there is an btrfs file system, which is then exported via NFS. Now all of the TCP locks will be used twice for two different instances, once for the TCP connection for the network block device, and then for the NFS export.

Quoted 6 lines

How exactly are we supposed to classify the locks to make it all work?

Or the loop device built on top of an ext4 file system which on a LVM/device mapper device. And suppose the loop device is then layered with a dm-error device for regression testing, and with another ext4 file system on top of that?

Ditto.

Quoted 5 lines

How exactly are we supposed to classify the locks in that situation? Where's the documentation and tutorials which explain how to make this work, if the responsibility is going to be dumped on the subsystem maintainers' laps? Or if the lockdep maintainers are expected to fix and classify all of these locks, are you volunteering to do this?

I have the will. I will.

Quoted 7 lines

How hard is it exactly to do all of this classification work, no matter whose responsibility it will ultimately be?

And if the answer is that it is too hard, then let me gently suggest to you that perhaps, if this is a case, that maybe this is a fundamental and fatal flaw with the cross-release and completion lockdep feature?

I don't understand this.

Quoted 3 lines

Best regards,

- Ted

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer
#5 lore raw
In reply to Theodore Ts'o
Quoted 4 lines · Theodore Ts'o

interpreted this as the lockdep maintainers saying, "hey, not my fault, it's the subsystem maintainer's fault for not properly classifying the locks" --- and thus dumping the responsibility in the subsystem maintainers' laps.

Let me clarify that I (as lockdep maintainer) disagree with that sentiment. I have spend a lot of time over the years staring at random code trying to fix lockdep splats. Its awesome if corresponding subsystem maintainers help out and many have, but I very much do not agree its their problem and their problem alone.

This attitude is one of the biggest issues I have with the crossrelease stuff and why I don't disagree with Ingo taking it out (for now).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Theodore Ts'o

interpreted this as the lockdep maintainers saying, "hey, not my fault, it's the subsystem maintainer's fault for not properly classifying the locks" --- and thus dumping the responsibility in the subsystem maintainers' laps.

Let me clarify that I (as lockdep maintainer) disagree with that sentiment. I have spend a lot of time over the years staring at random code trying to fix lockdep splats. Its awesome if corresponding subsystem maintainers help out and many have, but I very much do not agree its their problem and their problem alone.

I apologize to all of you. That's really not what I intended to say.

I said that other folks can annotate it for the sub-system better than lockdep developer, so suggested to invalidate locks making trouble and wanting to avoid annotating it at the moment, and validate those back when necessary with additional annotations.

It's my fault. I'm not sure how I should express what I want to say, but, I didn't intend to charge the responsibility to other folks.

Ideally, I think it's best to solve it with co-work. I should've been more careful to say that.

Again, I apologize for that, to lockdep and fs maintainers.

Of course, for cross-release, I have the will to annotate it or find a better way to avoid false positives. And I think I have to.

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw

PATCH 1/1 locking/lockdep: Remove the cross-release locking checks

In reply to Byungchul Park
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

Just to give a full context to everyone: the patch that removes the cross-release locking checks was Cc:-ed to lkml, I've attached the patch below again.

In general, as described in the changelog, the cross-release checks were historically just too painful (first they were too slow, and they also had a lot of false positives), and today, 4 months after its introduction, the cross-release checks *still* produce numerous false positives, especially in the filesystem space, but the continuous-integration testing folks were still having trouble with kthread locking patterns causing false positives:

https://bugs.freedesktop.org/show_bug.cgi?id=103950

which were resulting in two bad reactions:

- turning off lockdep

- writing patches that uglified unrelated subsystems

So while I appreciate the fixes that resulted from running cross-release, there's still numerous false positives, months after its interaction, which is unacceptable. For us to have this feature it has to have roughly similar qualities as compiler warnings:

- there's a "zero false positive warnings" policy

- plus any widespread changes to avoid warnings has to improve the code, not make it uglier.

Lockdep itself is a following that policy: the default state is that it produces no warnings upstream, and any annotations added to unrelated code documents the locking hierarchies.

While technically we could keep the cross-release checking code upstream and turn it off by default via the Kconfig switch, I'm not a big believer in such a policy for complex debugging code:

- We already did that for v4.14, two months ago:

b483cf3bc249: locking/lockdep: Disable cross-release features for now

... and re-enabled it for v4.15 - but the false positives are still not fixed.

- either the cross-release checking code can be fixed and then having it off by default is just wrong, because we can apply the fixed code again once it's fixed.

- or it cannot be fixed (or we don't have the manpower/interest to fix it), in which case having it off is only delaying the inevitable.

In any case, for v4.15 it's clear that the false positives are too numerous.

Thanks,

Ingo

=============================>

Quoted 1 line

From e966eaeeb623f09975ef362c2866fae6f86844f9 Mon Sep 17 00:00:00 2001

From: Ingo Molnar <[email protected]>
Date: Tue, 12 Dec 2017 12:31:16 +0100
Subject: [PATCH] locking/lockdep: Remove the cross-release locking checks

This code (CONFIG_LOCKDEP_CROSSRELEASE=y and CONFIG_LOCKDEP_COMPLETIONS=y), while it found a number of old bugs initially, was also causing too many false positives that caused people to disable lockdep - which is arguably a worse overall outcome.

If we disable cross-release by default but keep the code upstream then in practice the most likely outcome is that we'll allow the situation to degrade gradually, by allowing entropy to introduce more and more false positives, until it overwhelms maintenance capacity.

Another bad side effect was that people were trying to work around the false positives by uglifying/complicating unrelated code. There's a marked difference between annotating locking operations and uglifying good code just due to bad lock debugging code ...

This gradual decrease in quality happened to a number of debugging facilities in the kernel, and lockdep is pretty complex already, so we cannot risk this outcome.

Either cross-release checking can be done right with no false positives, or it should not be included in the upstream kernel.

( Note that it might make sense to maintain it out of tree and go through
the false positives every now and then and see whether new bugs were introduced. )

Cc Byungchul Park <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 Documentation/locking/crossrelease.txt | 874 ---------------------------------
 include/linux/completion.h             |  45 --
 include/linux/lockdep.h                | 125 -----
 include/linux/sched.h                  |  11 -
 kernel/locking/lockdep.c               | 652 ++----------------------
 lib/Kconfig.debug                      |  33 --
 6 files changed, 35 insertions(+), 1705 deletions(-)
Documentation/locking/crossrelease.txt deleted +0 -874
deleted file mode 100644
index bdf1423d5f99..000000000000
@@ -1,874 +0,0 @@
-Crossrelease
-============
-
-Started by Byungchul Park <[email protected]>
-
-Contents:
-
- (*) Background
-
-     - What causes deadlock
-     - How lockdep works
-
- (*) Limitation
-
-     - Limit lockdep
-     - Pros from the limitation
-     - Cons from the limitation
-     - Relax the limitation
-
- (*) Crossrelease
-
-     - Introduce crossrelease
-     - Introduce commit
-
- (*) Implementation
-
-     - Data structures
-     - How crossrelease works
-
- (*) Optimizations
-
-     - Avoid duplication
-     - Lockless for hot paths
-
- (*) APPENDIX A: What lockdep does to work aggresively
-
- (*) APPENDIX B: How to avoid adding false dependencies
-
-
-==========
-Background
-==========
-
-What causes deadlock
---------------------
-
-A deadlock occurs when a context is waiting for an event to happen,
-which is impossible because another (or the) context who can trigger the
-event is also waiting for another (or the) event to happen, which is
-also impossible due to the same reason.
-
-For example:
-
-   A context going to trigger event C is waiting for event A to happen.
-   A context going to trigger event A is waiting for event B to happen.
-   A context going to trigger event B is waiting for event C to happen.
-
-A deadlock occurs when these three wait operations run at the same time,
-because event C cannot be triggered if event A does not happen, which in
-turn cannot be triggered if event B does not happen, which in turn
-cannot be triggered if event C does not happen. After all, no event can
-be triggered since any of them never meets its condition to wake up.
-
-A dependency might exist between two waiters and a deadlock might happen
-due to an incorrect releationship between dependencies. Thus, we must
-define what a dependency is first. A dependency exists between them if:
-
-   1. There are two waiters waiting for each event at a given time.
-   2. The only way to wake up each waiter is to trigger its event.
-   3. Whether one can be woken up depends on whether the other can.
-
-Each wait in the example creates its dependency like:
-
-   Event C depends on event A.
-   Event A depends on event B.
-   Event B depends on event C.
-
-   NOTE: Precisely speaking, a dependency is one between whether a
-   waiter for an event can be woken up and whether another waiter for
-   another event can be woken up. However from now on, we will describe
-   a dependency as if it's one between an event and another event for
-   simplicity.
-
-And they form circular dependencies like:
-
-    -> C -> A -> B -
-   /                \
-   \                /
-    ----------------
-
-   where 'A -> B' means that event A depends on event B.
-
-Such circular dependencies lead to a deadlock since no waiter can meet
-its condition to wake up as described.
-
-CONCLUSION
-
-Circular dependencies cause a deadlock.
-
-
-How lockdep works
------------------
-
-Lockdep tries to detect a deadlock by checking dependencies created by
-lock operations, acquire and release. Waiting for a lock corresponds to
-waiting for an event, and releasing a lock corresponds to triggering an
-event in the previous section.
-
-In short, lockdep does:
-
-   1. Detect a new dependency.
-   2. Add the dependency into a global graph.
-   3. Check if that makes dependencies circular.
-   4. Report a deadlock or its possibility if so.
-
-For example, consider a graph built by lockdep that looks like:
-
-   A -> B -
-           \
-            -> E
-           /
-   C -> D -
-
-   where A, B,..., E are different lock classes.
-
-Lockdep will add a dependency into the graph on detection of a new
-dependency. For example, it will add a dependency 'E -> C' when a new
-dependency between lock E and lock C is detected. Then the graph will be:
-
-       A -> B -
-               \
-                -> E -
-               /      \
-    -> C -> D -        \
-   /                   /
-   \                  /
-    ------------------
-
-   where A, B,..., E are different lock classes.
-
-This graph contains a subgraph which demonstrates circular dependencies:
-
-                -> E -
-               /      \
-    -> C -> D -        \
-   /                   /
-   \                  /
-    ------------------
-
-   where C, D and E are different lock classes.
-
-This is the condition under which a deadlock might occur. Lockdep
-reports it on detection after adding a new dependency. This is the way
-how lockdep works.
-
-CONCLUSION
-
-Lockdep detects a deadlock or its possibility by checking if circular
-dependencies were created after adding each new dependency.
-
-
-==========
-Limitation
-==========
-
-Limit lockdep
--------------
-
-Limiting lockdep to work on only typical locks e.g. spin locks and
-mutexes, which are released within the acquire context, the
-implementation becomes simple but its capacity for detection becomes
-limited. Let's check pros and cons in next section.
-
-
-Pros from the limitation
-------------------------
-
-Given the limitation, when acquiring a lock, locks in a held_locks
-cannot be released if the context cannot acquire it so has to wait to
-acquire it, which means all waiters for the locks in the held_locks are
-stuck. It's an exact case to create dependencies between each lock in
-the held_locks and the lock to acquire.
-
-For example:
-
-   CONTEXT X
-   ---------
-   acquire A
-   acquire B /* Add a dependency 'A -> B' */
-   release B
-   release A
-
-   where A and B are different lock classes.
-
-When acquiring lock A, the held_locks of CONTEXT X is empty thus no
-dependency is added. But when acquiring lock B, lockdep detects and adds
-a new dependency 'A -> B' between lock A in the held_locks and lock B.
-They can be simply added whenever acquiring each lock.
-
-And data required by lockdep exists in a local structure, held_locks
-embedded in task_struct. Forcing to access the data within the context,
-lockdep can avoid racy problems without explicit locks while handling
-the local data.
-
-Lastly, lockdep only needs to keep locks currently being held, to build
-a dependency graph. However, relaxing the limitation, it needs to keep
-even locks already released, because a decision whether they created
-dependencies might be long-deferred.
-
-To sum up, we can expect several advantages from the limitation:
-
-   1. Lockdep can easily identify a dependency when acquiring a lock.
-   2. Races are avoidable while accessing local locks in a held_locks.
-   3. Lockdep only needs to keep locks currently being held.
-
-CONCLUSION
-
-Given the limitation, the implementation becomes simple and efficient.
-
-
-Cons from the limitation
-------------------------
-
-Given the limitation, lockdep is applicable only to typical locks. For
-example, page locks for page access or completions for synchronization
-cannot work with lockdep.
-
-Can we detect deadlocks below, under the limitation?
-
-Example 1:
-
-   CONTEXT X	   CONTEXT Y	   CONTEXT Z
-   ---------	   ---------	   ----------
-		   mutex_lock A
-   lock_page B
-		   lock_page B
-				   mutex_lock A /* DEADLOCK */
-				   unlock_page B held by X
-		   unlock_page B
-		   mutex_unlock A
-				   mutex_unlock A
-
-   where A and B are different lock classes.
-
-No, we cannot.
-
-Example 2:
-
-   CONTEXT X		   CONTEXT Y
-   ---------		   ---------
-			   mutex_lock A
-   mutex_lock A
-			   wait_for_complete B /* DEADLOCK */
-   complete B
-			   mutex_unlock A
-   mutex_unlock A
-
-   where A is a lock class and B is a completion variable.
-
-No, we cannot.
-
-CONCLUSION
-
-Given the limitation, lockdep cannot detect a deadlock or its
-possibility caused by page locks or completions.
-
-
-Relax the limitation
---------------------
-
-Under the limitation, things to create dependencies are limited to
-typical locks. However, synchronization primitives like page locks and
-completions, which are allowed to be released in any context, also
-create dependencies and can cause a deadlock. So lockdep should track
-these locks to do a better job. We have to relax the limitation for
-these locks to work with lockdep.
-
-Detecting dependencies is very important for lockdep to work because
-adding a dependency means adding an opportunity to check whether it
-causes a deadlock. The more lockdep adds dependencies, the more it
-thoroughly works. Thus Lockdep has to do its best to detect and add as
-many true dependencies into a graph as possible.
-
-For example, considering only typical locks, lockdep builds a graph like:
-
-   A -> B -
-           \
-            -> E
-           /
-   C -> D -
-
-   where A, B,..., E are different lock classes.
-
-On the other hand, under the relaxation, additional dependencies might
-be created and added. Assuming additional 'FX -> C' and 'E -> GX' are
-added thanks to the relaxation, the graph will be:
-
-         A -> B -
-                 \
-                  -> E -> GX
-                 /
-   FX -> C -> D -
-
-   where A, B,..., E, FX and GX are different lock classes, and a suffix
-   'X' is added on non-typical locks.
-
-The latter graph gives us more chances to check circular dependencies
-than the former. However, it might suffer performance degradation since
-relaxing the limitation, with which design and implementation of lockdep
-can be efficient, might introduce inefficiency inevitably. So lockdep
-should provide two options, strong detection and efficient detection.
-
-Choosing efficient detection:
-
-   Lockdep works with only locks restricted to be released within the
-   acquire context. However, lockdep works efficiently.
-
-Choosing strong detection:
-
-   Lockdep works with all synchronization primitives. However, lockdep
-   suffers performance degradation.
-
-CONCLUSION
-
-Relaxing the limitation, lockdep can add additional dependencies giving
-additional opportunities to check circular dependencies.
-
-
-============
-Crossrelease
-============
-
-Introduce crossrelease
-----------------------
-
-In order to allow lockdep to handle additional dependencies by what
-might be released in any context, namely 'crosslock', we have to be able
-to identify those created by crosslocks. The proposed 'crossrelease'
-feature provoides a way to do that.
-
-Crossrelease feature has to do:
-
-   1. Identify dependencies created by crosslocks.
-   2. Add the dependencies into a dependency graph.
-
-That's all. Once a meaningful dependency is added into graph, then
-lockdep would work with the graph as it did. The most important thing
-crossrelease feature has to do is to correctly identify and add true
-dependencies into the global graph.
-
-A dependency e.g. 'A -> B' can be identified only in the A's release
-context because a decision required to identify the dependency can be
-made only in the release context. That is to decide whether A can be
-released so that a waiter for A can be woken up. It cannot be made in
-other than the A's release context.
-
-It's no matter for typical locks because each acquire context is same as
-its release context, thus lockdep can decide whether a lock can be
-released in the acquire context. However for crosslocks, lockdep cannot
-make the decision in the acquire context but has to wait until the
-release context is identified.
-
-Therefore, deadlocks by crosslocks cannot be detected just when it
-happens, because those cannot be identified until the crosslocks are
-released. However, deadlock possibilities can be detected and it's very
-worth. See 'APPENDIX A' section to check why.
-
-CONCLUSION
-
-Using crossrelease feature, lockdep can work with what might be released
-in any context, namely crosslock.
-
-
-Introduce commit
-----------------
-
-Since crossrelease defers the work adding true dependencies of
-crosslocks until they are actually released, crossrelease has to queue
-all acquisitions which might create dependencies with the crosslocks.
-Then it identifies dependencies using the queued data in batches at a
-proper time. We call it 'commit'.
-
-There are four types of dependencies:
-
-1. TT type: 'typical lock A -> typical lock B'
-
-   Just when acquiring B, lockdep can see it's in the A's release
-   context. So the dependency between A and B can be identified
-   immediately. Commit is unnecessary.
-
-2. TC type: 'typical lock A -> crosslock BX'
-
-   Just when acquiring BX, lockdep can see it's in the A's release
-   context. So the dependency between A and BX can be identified
-   immediately. Commit is unnecessary, too.
-
-3. CT type: 'crosslock AX -> typical lock B'
-
-   When acquiring B, lockdep cannot identify the dependency because
-   there's no way to know if it's in the AX's release context. It has
-   to wait until the decision can be made. Commit is necessary.
-
-4. CC type: 'crosslock AX -> crosslock BX'
-
-   When acquiring BX, lockdep cannot identify the dependency because
-   there's no way to know if it's in the AX's release context. It has
-   to wait until the decision can be made. Commit is necessary.
-   But, handling CC type is not implemented yet. It's a future work.
-
-Lockdep can work without commit for typical locks, but commit step is
-necessary once crosslocks are involved. Introducing commit, lockdep
-performs three steps. What lockdep does in each step is:
-
-1. Acquisition: For typical locks, lockdep does what it originally did
-   and queues the lock so that CT type dependencies can be checked using
-   it at the commit step. For crosslocks, it saves data which will be
-   used at the commit step and increases a reference count for it.
-
-2. Commit: No action is reauired for typical locks. For crosslocks,
-   lockdep adds CT type dependencies using the data saved at the
-   acquisition step.
-
-3. Release: No changes are required for typical locks. When a crosslock
-   is released, it decreases a reference count for it.
-
-CONCLUSION
-
-Crossrelease introduces commit step to handle dependencies of crosslocks
-in batches at a proper time.
-
-
-==============
-Implementation
-==============
-
-Data structures
----------------
-
-Crossrelease introduces two main data structures.
-
-1. hist_lock
-
-   This is an array embedded in task_struct, for keeping lock history so
-   that dependencies can be added using them at the commit step. Since
-   it's local data, it can be accessed locklessly in the owner context.
-   The array is filled at the acquisition step and consumed at the
-   commit step. And it's managed in circular manner.
-
-2. cross_lock
-
-   One per lockdep_map exists. This is for keeping data of crosslocks
-   and used at the commit step.
-
-
-How crossrelease works
-----------------------
-
-It's the key of how crossrelease works, to defer necessary works to an
-appropriate point in time and perform in at once at the commit step.
-Let's take a look with examples step by step, starting from how lockdep
-works without crossrelease for typical locks.
-
-   acquire A /* Push A onto held_locks */
-   acquire B /* Push B onto held_locks and add 'A -> B' */
-   acquire C /* Push C onto held_locks and add 'B -> C' */
-   release C /* Pop C from held_locks */
-   release B /* Pop B from held_locks */
-   release A /* Pop A from held_locks */
-
-   where A, B and C are different lock classes.
-
-   NOTE: This document assumes that readers already understand how
-   lockdep works without crossrelease thus omits details. But there's
-   one thing to note. Lockdep pretends to pop a lock from held_locks
-   when releasing it. But it's subtly different from the original pop
-   operation because lockdep allows other than the top to be poped.
-
-In this case, lockdep adds 'the top of held_locks -> the lock to acquire'
-dependency every time acquiring a lock.
-
-After adding 'A -> B', a dependency graph will be:
-
-   A -> B
-
-   where A and B are different lock classes.
-
-And after adding 'B -> C', the graph will be:
-
-   A -> B -> C
-
-   where A, B and C are different lock classes.
-
-Let's performs commit step even for typical locks to add dependencies.
-Of course, commit step is not necessary for them, however, it would work
-well because this is a more general way.
-
-   acquire A
-   /*
-    * Queue A into hist_locks
-    *
-    * In hist_locks: A
-    * In graph: Empty
-    */
-
-   acquire B
-   /*
-    * Queue B into hist_locks
-    *
-    * In hist_locks: A, B
-    * In graph: Empty
-    */
-
-   acquire C
-   /*
-    * Queue C into hist_locks
-    *
-    * In hist_locks: A, B, C
-    * In graph: Empty
-    */
-
-   commit C
-   /*
-    * Add 'C -> ?'
-    * Answer the following to decide '?'
-    * What has been queued since acquire C: Nothing
-    *
-    * In hist_locks: A, B, C
-    * In graph: Empty
-    */
-
-   release C
-
-   commit B
-   /*
-    * Add 'B -> ?'
-    * Answer the following to decide '?'
-    * What has been queued since acquire B: C
-    *
-    * In hist_locks: A, B, C
-    * In graph: 'B -> C'
-    */
-
-   release B
-
-   commit A
-   /*
-    * Add 'A -> ?'
-    * Answer the following to decide '?'
-    * What has been queued since acquire A: B, C
-    *
-    * In hist_locks: A, B, C
-    * In graph: 'B -> C', 'A -> B', 'A -> C'
-    */
-
-   release A
-
-   where A, B and C are different lock classes.
-
-In this case, dependencies are added at the commit step as described.
-
-After commits for A, B and C, the graph will be:
-
-   A -> B -> C
-
-   where A, B and C are different lock classes.
-
-   NOTE: A dependency 'A -> C' is optimized out.
-
-We can see the former graph built without commit step is same as the
-latter graph built using commit steps. Of course the former way leads to
-earlier finish for building the graph, which means we can detect a
-deadlock or its possibility sooner. So the former way would be prefered
-when possible. But we cannot avoid using the latter way for crosslocks.
-
-Let's look at how commit steps work for crosslocks. In this case, the
-commit step is performed only on crosslock AX as real. And it assumes
-that the AX release context is different from the AX acquire context.
-
-   BX RELEASE CONTEXT		   BX ACQUIRE CONTEXT
-   ------------------		   ------------------
-				   acquire A
-				   /*
-				    * Push A onto held_locks
-				    * Queue A into hist_locks
-				    *
-				    * In held_locks: A
-				    * In hist_locks: A
-				    * In graph: Empty
-				    */
-
-				   acquire BX
-				   /*
-				    * Add 'the top of held_locks -> BX'
-				    *
-				    * In held_locks: A
-				    * In hist_locks: A
-				    * In graph: 'A -> BX'
-				    */
-
-   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-   It must be guaranteed that the following operations are seen after
-   acquiring BX globally. It can be done by things like barrier.
-   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-   acquire C
-   /*
-    * Push C onto held_locks
-    * Queue C into hist_locks
-    *
-    * In held_locks: C
-    * In hist_locks: C
-    * In graph: 'A -> BX'
-    */
-
-   release C
-   /*
-    * Pop C from held_locks
-    *
-    * In held_locks: Empty
-    * In hist_locks: C
-    * In graph: 'A -> BX'
-    */
-				   acquire D
-				   /*
-				    * Push D onto held_locks
-				    * Queue D into hist_locks
-				    * Add 'the top of held_locks -> D'
-				    *
-				    * In held_locks: A, D
-				    * In hist_locks: A, D
-				    * In graph: 'A -> BX', 'A -> D'
-				    */
-   acquire E
-   /*
-    * Push E onto held_locks
-    * Queue E into hist_locks
-    *
-    * In held_locks: E
-    * In hist_locks: C, E
-    * In graph: 'A -> BX', 'A -> D'
-    */
-
-   release E
-   /*
-    * Pop E from held_locks
-    *
-    * In held_locks: Empty
-    * In hist_locks: D, E
-    * In graph: 'A -> BX', 'A -> D'
-    */
-				   release D
-				   /*
-				    * Pop D from held_locks
-				    *
-				    * In held_locks: A
-				    * In hist_locks: A, D
-				    * In graph: 'A -> BX', 'A -> D'
-				    */
-   commit BX
-   /*
-    * Add 'BX -> ?'
-    * What has been queued since acquire BX: C, E
-    *
-    * In held_locks: Empty
-    * In hist_locks: D, E
-    * In graph: 'A -> BX', 'A -> D',
-    *           'BX -> C', 'BX -> E'
-    */
-
-   release BX
-   /*
-    * In held_locks: Empty
-    * In hist_locks: D, E
-    * In graph: 'A -> BX', 'A -> D',
-    *           'BX -> C', 'BX -> E'
-    */
-				   release A
-				   /*
-				    * Pop A from held_locks
-				    *
-				    * In held_locks: Empty
-				    * In hist_locks: A, D
-				    * In graph: 'A -> BX', 'A -> D',
-				    *           'BX -> C', 'BX -> E'
-				    */
-
-   where A, BX, C,..., E are different lock classes, and a suffix 'X' is
-   added on crosslocks.
-
-Crossrelease considers all acquisitions after acqiuring BX are
-candidates which might create dependencies with BX. True dependencies
-will be determined when identifying the release context of BX. Meanwhile,
-all typical locks are queued so that they can be used at the commit step.
-And then two dependencies 'BX -> C' and 'BX -> E' are added at the
-commit step when identifying the release context.
-
-The final graph will be, with crossrelease:
-
-               -> C
-              /
-       -> BX -
-      /       \
-   A -         -> E
-      \
-       -> D
-
-   where A, BX, C,..., E are different lock classes, and a suffix 'X' is
-   added on crosslocks.
-
-However, the final graph will be, without crossrelease:
-
-   A -> D
-
-   where A and D are different lock classes.
-
-The former graph has three more dependencies, 'A -> BX', 'BX -> C' and
-'BX -> E' giving additional opportunities to check if they cause
-deadlocks. This way lockdep can detect a deadlock or its possibility
-caused by crosslocks.
-
-CONCLUSION
-
-We checked how crossrelease works with several examples.
-
-
-=============
-Optimizations
-=============
-
-Avoid duplication
------------------
-
-Crossrelease feature uses a cache like what lockdep already uses for
-dependency chains, but this time it's for caching CT type dependencies.
-Once that dependency is cached, the same will never be added again.
-
-
-Lockless for hot paths
-----------------------
-
-To keep all locks for later use at the commit step, crossrelease adopts
-a local array embedded in task_struct, which makes access to the data
-lockless by forcing it to happen only within the owner context. It's
-like how lockdep handles held_locks. Lockless implmentation is important
-since typical locks are very frequently acquired and released.
-
-
-=================================================
-APPENDIX A: What lockdep does to work aggresively
-=================================================
-
-A deadlock actually occurs when all wait operations creating circular
-dependencies run at the same time. Even though they don't, a potential
-deadlock exists if the problematic dependencies exist. Thus it's
-meaningful to detect not only an actual deadlock but also its potential
-possibility. The latter is rather valuable. When a deadlock occurs
-actually, we can identify what happens in the system by some means or
-other even without lockdep. However, there's no way to detect possiblity
-without lockdep unless the whole code is parsed in head. It's terrible.
-Lockdep does the both, and crossrelease only focuses on the latter.
-
-Whether or not a deadlock actually occurs depends on several factors.
-For example, what order contexts are switched in is a factor. Assuming
-circular dependencies exist, a deadlock would occur when contexts are
-switched so that all wait operations creating the dependencies run
-simultaneously. Thus to detect a deadlock possibility even in the case
-that it has not occured yet, lockdep should consider all possible
-combinations of dependencies, trying to:
-
-1. Use a global dependency graph.
-
-   Lockdep combines all dependencies into one global graph and uses them,
-   regardless of which context generates them or what order contexts are
-   switched in. Aggregated dependencies are only considered so they are
-   prone to be circular if a problem exists.
-
-2. Check dependencies between classes instead of instances.
-
-   What actually causes a deadlock are instances of lock. However,
-   lockdep checks dependencies between classes instead of instances.
-   This way lockdep can detect a deadlock which has not happened but
-   might happen in future by others but the same class.
-
-3. Assume all acquisitions lead to waiting.
-
-   Although locks might be acquired without waiting which is essential
-   to create dependencies, lockdep assumes all acquisitions lead to
-   waiting since it might be true some time or another.
-
-CONCLUSION
-
-Lockdep detects not only an actual deadlock but also its possibility,
-and the latter is more valuable.
-
-
-==================================================
-APPENDIX B: How to avoid adding false dependencies
-==================================================
-
-Remind what a dependency is. A dependency exists if:
-
-   1. There are two waiters waiting for each event at a given time.
-   2. The only way to wake up each waiter is to trigger its event.
-   3. Whether one can be woken up depends on whether the other can.
-
-For example:
-
-   acquire A
-   acquire B /* A dependency 'A -> B' exists */
-   release B
-   release A
-
-   where A and B are different lock classes.
-
-A depedency 'A -> B' exists since:
-
-   1. A waiter for A and a waiter for B might exist when acquiring B.
-   2. Only way to wake up each is to release what it waits for.
-   3. Whether the waiter for A can be woken up depends on whether the
-      other can. IOW, TASK X cannot release A if it fails to acquire B.
-
-For another example:
-
-   TASK X			   TASK Y
-   ------			   ------
-				   acquire AX
-   acquire B /* A dependency 'AX -> B' exists */
-   release B
-   release AX held by Y
-
-   where AX and B are different lock classes, and a suffix 'X' is added
-   on crosslocks.
-
-Even in this case involving crosslocks, the same rule can be applied. A
-depedency 'AX -> B' exists since:
-
-   1. A waiter for AX and a waiter for B might exist when acquiring B.
-   2. Only way to wake up each is to release what it waits for.
-   3. Whether the waiter for AX can be woken up depends on whether the
-      other can. IOW, TASK X cannot release AX if it fails to acquire B.
-
-Let's take a look at more complicated example:
-
-   TASK X			   TASK Y
-   ------			   ------
-   acquire B
-   release B
-   fork Y
-				   acquire AX
-   acquire C /* A dependency 'AX -> C' exists */
-   release C
-   release AX held by Y
-
-   where AX, B and C are different lock classes, and a suffix 'X' is
-   added on crosslocks.
-
-Does a dependency 'AX -> B' exist? Nope.
-
-Two waiters are essential to create a dependency. However, waiters for
-AX and B to create 'AX -> B' cannot exist at the same time in this
-example. Thus the dependency 'AX -> B' cannot be created.
-
-It would be ideal if the full set of true ones can be considered. But
-we can ensure nothing but what actually happened. Relying on what
-actually happens at runtime, we can anyway add only true ones, though
-they might be a subset of true ones. It's similar to how lockdep works
-for typical locks. There might be more true dependencies than what
-lockdep has detected in runtime. Lockdep has no choice but to rely on
-what actually happens. Crossrelease also relies on it.
-
-CONCLUSION
-
-Relying on what actually happens, lockdep can avoid adding false
-dependencies.
include/linux/completion.h +0 -45

67 lines not on this page · raw message

include/linux/lockdep.h +0 -125

151 lines not on this page · raw message

include/linux/sched.h +0 -11
index 21991d668d35..9ce6c3001e9f 100644
@@ -849,17 +849,6 @@ struct task_struct {
 	struct held_lock		held_locks[MAX_LOCK_DEPTH];
 #endif
 
-#ifdef CONFIG_LOCKDEP_CROSSRELEASE
-#define MAX_XHLOCKS_NR 64UL
-	struct hist_lock *xhlocks; /* Crossrelease history locks */
-	unsigned int xhlock_idx;
-	/* For restoring at history boundaries */
-	unsigned int xhlock_idx_hist[XHLOCK_CTX_NR];
-	unsigned int hist_id;
-	/* For overwrite check at each context exit */
-	unsigned int hist_id_save[XHLOCK_CTX_NR];
-#endif
-
 #ifdef CONFIG_UBSAN
 	unsigned int			in_ubsan;
 #endif
kernel/locking/lockdep.c +35 -617

746 lines not on this page · raw message

lib/Kconfig.debug +0 -33
index 947d3e2ed5c2..9d5b78aad4c5 100644
@@ -1099,8 +1099,6 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE
-	select LOCKDEP_COMPLETIONS
 	select TRACE_IRQFLAGS
 	default n
 	help
@@ -1170,37 +1168,6 @@ config LOCK_STAT
 	 CONFIG_LOCK_STAT defines "contended" and "acquired" lock events.
 	 (CONFIG_LOCKDEP defines "acquire" and "release" events.)
 
-config LOCKDEP_CROSSRELEASE
-	bool
-	help
-	 This makes lockdep work for crosslock which is a lock allowed to
-	 be released in a different context from the acquisition context.
-	 Normally a lock must be released in the context acquiring the lock.
-	 However, relexing this constraint helps synchronization primitives
-	 such as page locks or completions can use the lock correctness
-	 detector, lockdep.
-
-config LOCKDEP_COMPLETIONS
-	bool
-	help
-	 A deadlock caused by wait_for_completion() and complete() can be
-	 detected by lockdep using crossrelease feature.
-
-config BOOTPARAM_LOCKDEP_CROSSRELEASE_FULLSTACK
-	bool "Enable the boot parameter, crossrelease_fullstack"
-	depends on LOCKDEP_CROSSRELEASE
-	default n
-	help
-	 The lockdep "cross-release" feature needs to record stack traces
-	 (of calling functions) for all acquisitions, for eventual later
-	 use during analysis. By default only a single caller is recorded,
-	 because the unwind operation can be very expensive with deeper
-	 stack chains.
-
-	 However a boot parameter, crossrelease_fullstack, was
-	 introduced since sometimes deeper traces are required for full
-	 analysis. This option turns on the boot parameter.
-
 config DEBUG_LOCKDEP
 	bool "Lock dependency engine debugging"
 	depends on DEBUG_KERNEL && LOCKDEP
#8 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 33 lines · Ingo Molnar
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

Just to give a full context to everyone: the patch that removes the cross-release locking checks was Cc:-ed to lkml, I've attached the patch below again.

Quoted 6 lines

In general, as described in the changelog, the cross-release checks were historically just too painful (first they were too slow, and they also had a lot of false positives), and today, 4 months after its introduction, the cross-release checks *still* produce numerous false positives, especially in the filesystem space, but the continuous-integration testing folks were still having trouble with kthread locking patterns causing false positives:

I admit false positives are the main problem, that should be solved.

I'm going willingly to try my best to solve that. However, as you may know through introduction of lockdep, it's not something that I can do easily and shortly on my own. It need take time to annotate properly to avoid false positives.

Quoted 7 lines

https://bugs.freedesktop.org/show_bug.cgi?id=103950

which were resulting in two bad reactions:

- turning off lockdep

- writing patches that uglified unrelated subsystems

I can't give you a solution at the moment but it's something we think more so that we classify locks properly and not uglify them.

Even without cross-release, once we start to add lock_acquire() in submit_bio_wait() in the ugly way to consider wait_for_completion() someday, we would face this problem again. It's not an easy problem, however, it's worth trying.

Quoted 6 lines

So while I appreciate the fixes that resulted from running cross-release, there's still numerous false positives, months after its interaction, which is unacceptable. For us to have this feature it has to have roughly similar qualities as compiler warnings:

- there's a "zero false positive warnings" policy

It's almost impossible... but need time. I wonder if lockdep did at the beginning. If I can, I want to fix false positive as many as possible by myself. But, unluckily it does not happen in my machine. I want to get informed from others, keeping it in mainline tree.

Quoted 2 lines

- plus any widespread changes to avoid warnings has to improve the code, not make it uglier.

Agree.

Quoted 11 lines

Lockdep itself is a following that policy: the default state is that it produces no warnings upstream, and any annotations added to unrelated code documents the locking hierarchies.

While technically we could keep the cross-release checking code upstream and turn it off by default via the Kconfig switch, I'm not a big believer in such a policy for complex debugging code:

- We already did that for v4.14, two months ago:

b483cf3bc249: locking/lockdep: Disable cross-release features for now

The main reason disabling it was "performance regression".

Quoted 2 lines

... and re-enabled it for v4.15 - but the false positives are still not fixed.

Right. But, all false positives cannot be fixed in a short period. We need time to annotate them one by one.

Quoted 1 line

- either the cross-release checking code can be fixed and then having it off by

It's not a problem of cross-release checking code. The way I have to fix it should be to add additional annotation or change the way to assign lock classes.

Quoted 5 lines

default is just wrong, because we can apply the fixed code again once it's fixed.

- or it cannot be fixed (or we don't have the manpower/interest to fix it), in which case having it off is only delaying the inevitable.

The more precisely assigning lock classes, the more false positives would be getting fixed. It's not something messing it as time goes.

Quoted 7 lines

In any case, for v4.15 it's clear that the false positives are too numerous.

Thanks,

Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 107 lines · Byungchul Park
Quoted 33 lines · Ingo Molnar
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

Just to give a full context to everyone: the patch that removes the cross-release locking checks was Cc:-ed to lkml, I've attached the patch below again.

Quoted 6 lines

In general, as described in the changelog, the cross-release checks were historically just too painful (first they were too slow, and they also had a lot of false positives), and today, 4 months after its introduction, the cross-release checks *still* produce numerous false positives, especially in the filesystem space, but the continuous-integration testing folks were still having trouble with kthread locking patterns causing false positives:

I admit false positives are the main problem, that should be solved.

I'm going willingly to try my best to solve that. However, as you may know through introduction of lockdep, it's not something that I can do easily and shortly on my own. It need take time to annotate properly to avoid false positives.

Quoted 7 lines

https://bugs.freedesktop.org/show_bug.cgi?id=103950

which were resulting in two bad reactions:

- turning off lockdep

- writing patches that uglified unrelated subsystems

I can't give you a solution at the moment but it's something we think more so that we classify locks properly and not uglify them.

Even without cross-release, once we start to add lock_acquire() in submit_bio_wait() in the ugly way to consider wait_for_completion() someday, we would face this problem again. It's not an easy problem, however, it's worth trying.

Quoted 6 lines

So while I appreciate the fixes that resulted from running cross-release, there's still numerous false positives, months after its interaction, which is unacceptable. For us to have this feature it has to have roughly similar qualities as compiler warnings:

- there's a "zero false positive warnings" policy

It's almost impossible... but need time. I wonder if lockdep did at the beginning. If I can, I want to fix false positive as many as possible by myself. But, unluckily it does not happen in my machine. I want to get informed from others, keeping it in mainline tree.

Quoted 2 lines

- plus any widespread changes to avoid warnings has to improve the code, not make it uglier.

Agree.

Quoted 11 lines

Lockdep itself is a following that policy: the default state is that it produces no warnings upstream, and any annotations added to unrelated code documents the locking hierarchies.

While technically we could keep the cross-release checking code upstream and turn it off by default via the Kconfig switch, I'm not a big believer in such a policy for complex debugging code:

- We already did that for v4.14, two months ago:

b483cf3bc249: locking/lockdep: Disable cross-release features for now

The main reason disabling it was "performance regression".

Quoted 2 lines

... and re-enabled it for v4.15 - but the false positives are still not fixed.

Right. But, all false positives cannot be fixed in a short period. We need time to annotate them one by one.

Quoted 1 line

- either the cross-release checking code can be fixed and then having it off by

It's not a problem of cross-release checking code. The way I have to fix it should be to add additional annotation or change the way to assign lock classes.

Quoted 16 lines
Quoted 5 lines

default is just wrong, because we can apply the fixed code again once it's fixed.

- or it cannot be fixed (or we don't have the manpower/interest to fix it), in which case having it off is only delaying the inevitable.

The more precisely assigning lock classes, the more false positives would be getting fixed. It's not something messing it as time goes.

Quoted 7 lines

In any case, for v4.15 it's clear that the false positives are too numerous.

Thanks,

Ingo

Hello all,

I'm against the removing, not only because it's my work.

We've already kept adding lock_acquire() manually to consider wait_for_completion() or general waiters one by one until now. It's certainly what we need.

Cross-release does it, but it makes trouble because it tries to consider all waiters *at one time* instead of one by one.

Yes, it's a obvious problem. Doesn't we consider an option, that is, to invalidate locks making trouble quickly and validate it back one by one, so that almost other waiters can still get benefit from cross-release?

Of course, it's not the ultimate solution. But, that would be much better than stopping all the benefit at once.

For example, in the case of fs issues, for now we can invalidate wait_for_completion() in submit_bio_wait() and re-validate it later, of course, I really want to find more fundamental solution though.

Is it possible?

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 2 lines · Byungchul Park

For example, in the case of fs issues, for now we can invalidate wait_for_completion() in submit_bio_wait()....

And this will spawn a checkpatch.pl ERROR:

    	      	    ERROR("LOCKDEP",
		    "lockdep_no_validate class is reserved for device->mutex.\n" . $herecurr);

This mention in checkpatch.pl is the only documentation I've been able to find about lockdep_set_novalidate_class(), by the way.

Quoted 2 lines

... and re-validate it later, of course, I really want to find more fundamental solution though.

Oh, and I was finally able to find the quote that the *only* people who are likely to be able to deal with lock annotations are the subsystem maintainers. But if the ways of dealing with lock annotations are not documented, such that subsystem maintainers are going to have a very hard time figuring out *how* to deal with it, it seems that lock classification as a solution to cross-release false positives seems.... unlikely:

From: Byungchul Park <[email protected]>
Date: Fri, 8 Dec 2017 18:27:45 +0900
Subject: Re: [PATCH v4 72/73] xfs: Convert mru cache to XArray

1) Firstly, it's hard to assign lock classes *properly*. By default, it relies on the caller site of lockdep_init_map(), but we need to assign another class manually, where ordering rules are complicated so cannot rely on the caller site. That *only* can be done by experts of the subsystem.

- Ted

#11 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 9 lines · Theodore Ts'o
Quoted 2 lines · Byungchul Park

For example, in the case of fs issues, for now we can invalidate wait_for_completion() in submit_bio_wait()....

And this will spawn a checkpatch.pl ERROR:

                    ERROR("LOCKDEP",
                    "lockdep_no_validate class is reserved for device->mutex.\n" . $herecurr);
Quoted 8 lines

This mention in checkpatch.pl is the only documentation I've been able to find about lockdep_set_novalidate_class(), by the way.

Quoted 2 lines

... and re-validate it later, of course, I really want to find more fundamental solution though.

Oh, and I was finally able to find the quote that the *only* people who are likely to be able to deal with lock annotations are the

Right. Using the word, "only", is one that I should not have done and I apologize for.

They are just "only" people who solve and classify locks quickly, assuming all of kernel guys are familiar with lockdep annotations. Thus, even this statement is bad as well, since no good document for that exists, as you pointed out. I agree.

Quoted 3 lines

subsystem maintainers. But if the ways of dealing with lock annotations are not documented, such that subsystem maintainers are going to have a very hard time figuring out *how* to deal with it, it

Right. I've agreed with this, since you pointed out it's problem not to be documented well.

Quoted 14 lines

seems that lock classification as a solution to cross-release false positives seems.... unlikely:

From: Byungchul Park <[email protected]>
Date: Fri, 8 Dec 2017 18:27:45 +0900
Subject: Re: [PATCH v4 72/73] xfs: Convert mru cache to XArray

1) Firstly, it's hard to assign lock classes *properly*. By default, it relies on the caller site of lockdep_init_map(), but we need to assign another class manually, where ordering rules are complicated so cannot rely on the caller site. That *only* can be done by experts of the subsystem.

- Ted

Signature · 2 lines
Thanks,
Byungchul
#12 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

In reply to Theodore Ts'o
Quoted 2 lines · Theodore Ts'o

seems that lock classification as a solution to cross-release false positives seems.... unlikely:

For this, let me explain more.

For example, either to use cross-release or to consider wait_for_completion() in submit_bio_wait() manually using lock_acquire() someday, classifying locks or waiters precisely is needed.

All locks should belong to one class if each path of acquisition can be switchable each other within the class at any time. Otherwise, they should belong to a different class.

Even though they are different classes but belong to one class roughly, no problem comes into view unless they are connected each other via extra dependency chains. But, once they get connected, we can see problems by the wrong classification. That can happen even w/o cross-release.

Of course, as you pointed out, cross-release generates many chains between classes, assuming all classes are well- classified. But, practically well-classifying is not an easy work.

So that's why I suggested the way since anyway that's better than removing all. If that's allowed, I can invalidate those waiters, using e.g. completion_init_nomap().

Quoted 11 lines

From: Byungchul Park <[email protected]>
Date: Fri, 8 Dec 2017 18:27:45 +0900
Subject: Re: [PATCH v4 72/73] xfs: Convert mru cache to XArray

1) Firstly, it's hard to assign lock classes *properly*. By default, it relies on the caller site of lockdep_init_map(), but we need to assign another class manually, where ordering rules are complicated so cannot rely on the caller site. That *only* can be done by experts of the subsystem.

- Ted

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 4 lines · Byungchul Park

All locks should belong to one class if each path of acquisition can be switchable each other within the class at any time. Otherwise, they should belong to a different class.

OK, so let's go back to my case of a Network Block Device with a local file system mounted on it, which is then exported via NFS.

So an incoming TCP packet can go into the NFS server subsystem, then be processed by local disk file system, which then does an I/O operation to the NBD device, which results in an TCP packet being sent out. Then the response will come back over TCP, into the NBD block layer, then into the local disk file system, and that will result in an outgoing response to the TCP connection for the NFS protocol.

In order to avoid cross release problems, all locks associated with the incoming TCP connection will need to be classified as belonging to a different class as the outgoing TCP connection. Correct? One solution might be to put every single TCP connection into a separate class --- but that will explode the number of lock classes which Lockdep will need to track, and there is a limited number of lock classes (set at compile time) that Lockdep can track. So if that doesn't work, we will have to put something ugly which manually makes certain TCP connections "magic" and require them to be put into a separate class than all other TCP connections, which will get collapsed into a single class. Basically, any TCP connection which is either originated by the kernel, or passed in from userspace into the kernel and used by some kernel subsystem, will have to be assigned its own lockdep class.

If the TCP connection gets closed, we don't need to track that lockdep class any more. (Or if a device mapper device is torn down, we similarly don't need any unique lockdep classes created for that device mapper device.) Is there a way to tell lockdep that a set of lockdep classes can be released so we can recover the kernel memory to be used to track some new TCP connection or some new device mapper device?

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

Quoted 31 lines · Theodore Ts'o
Quoted 4 lines · Byungchul Park

All locks should belong to one class if each path of acquisition can be switchable each other within the class at any time. Otherwise, they should belong to a different class.

OK, so let's go back to my case of a Network Block Device with a local file system mounted on it, which is then exported via NFS.

So an incoming TCP packet can go into the NFS server subsystem, then be processed by local disk file system, which then does an I/O operation to the NBD device, which results in an TCP packet being sent out. Then the response will come back over TCP, into the NBD block layer, then into the local disk file system, and that will result in an outgoing response to the TCP connection for the NFS protocol.

In order to avoid cross release problems, all locks associated with the incoming TCP connection will need to be classified as belonging to a different class as the outgoing TCP connection. Correct? One solution might be to put every single TCP connection into a separate class --- but that will explode the number of lock classes which Lockdep will need to track, and there is a limited number of lock classes (set at compile time) that Lockdep can track. So if that doesn't work, we will have to put something ugly which manually makes certain TCP connections "magic" and require them to be put into a separate class than all other TCP connections, which will get collapsed into a single class. Basically, any TCP connection which is either originated by the kernel, or passed in from userspace into the kernel and used by some kernel subsystem, will have to be assigned its own lockdep class.

Quoted 7 lines

If the TCP connection gets closed, we don't need to track that lockdep class any more. (Or if a device mapper device is torn down, we similarly don't need any unique lockdep classes created for that device mapper device.) Is there a way to tell lockdep that a set of lockdep classes can be released so we can recover the kernel memory to be used to track some new TCP connection or some new device mapper device?

Right. I also think lockdep should be able to reflect that kind of dynamic situations to do a better job.

The fact that kernel works well w/o that work doesn't mean current status is perfect, in my opinion.

As you know, lockdep is running within very limited environment so it's very hard to achieve it.

However, anyway, I think that's a problem and should be solved by modifying lockdep core. Actually, that had been one on my to-dos, if allowed.

For some waiters, for which this is only solution to play with cross-release, I think we can invalidate those waiters for now, while all others still get benefit.

We have added acquire annotations manually to consider waiters one by one, and I am sure it's going to continue in the future.

IMO, considering all waiters at once and fixing false positives in a right way or invalidating one by one is better than considering waiters one by one as is, of course, while keeping off by default.

Signature · 2 lines
Thanks,
Byungchul
#15 lore raw
In reply to Byungchul Park
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

I admit the cross-release feature had introduced several false positives about 4 times(?), maybe. And I suggested roughly 3 ways to solve it. I should have explained each in more detail. The lack might have led some to misunderstand.

(1) The best way: To classify all waiters correctly.

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

(3) The 3rd way: To invalidate waiters making trouble.

Of course, this is not the best. Now that you have already spent a lot of time to fix original lockdep's problems since lockdep was introduced in 2006, we don't need to use this way for typical locks except a few special cases. Lockdep is fairly robust by now.

And I understand you don't want to spend more time to fix additional problems again. Now that the situation is different from the time, 2006, it's not too bad to use this way to handle the issues.

IMO, the ways can be considered together at a time, which perhaps would be even better.

Talking about what Ingo said in the commit msg.. I want to ask him back, if he did it with no false positives at the moment merging it in 2006, without using (2) or (3) method. I bet he know what it means.. And classifying locks/waiters correctly is not something uglifying code but a way to document code better. I've felt ill at ease because of the unnatural and forced explanation.

--
Thanks,
Byungchul

#16 lore raw
Quoted 59 lines · Byungchul Park
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

I admit the cross-release feature had introduced several false positives about 4 times(?), maybe. And I suggested roughly 3 ways to solve it. I should have explained each in more detail. The lack might have led some to misunderstand.

(1) The best way: To classify all waiters correctly.

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

(3) The 3rd way: To invalidate waiters making trouble.

Of course, this is not the best. Now that you have already spent a lot of time to fix original lockdep's problems since lockdep was introduced in 2006, we don't need to use this way for typical locks except a few special cases. Lockdep is fairly robust by now.

Quoted 7 lines

And I understand you don't want to spend more time to fix additional problems again. Now that the situation is different from the time, 2006, it's not too bad to use this way to handle the issues.

IMO, the ways can be considered together at a time, which perhaps would be even better.

+cc [email protected]

Quoted 1 line

Talking about what Ingo said in the commit msg.. I want to ask him back,

I'm sorry for missing specifying the commit I'm talking about.

e966eaeeb locking/lockdep: Remove the cross-release locking checks

Quoted 9 lines

if he did it with no false positives at the moment merging it in 2006, without using (2) or (3) method. I bet he know what it means.. And classifying locks/waiters correctly is not something uglifying code but a way to document code better. I've felt ill at ease because of the unnatural and forced explanation.

--
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#17 lore raw
In reply to Byungchul Park
Quoted 2 lines · Byungchul Park

(1) The best way: To classify all waiters correctly.

It's really not all waiters, but all *locks*, no?

Quoted 8 lines

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

The problem is that it's not one subsystem, but *many*. And it's the interactions between the subsystems.

Consider the example I gave of a network block device, on which a local disk file system is mounted, which is then exported over NFS. So we have the Networking (TCP) stack involved, the NBD device driver, the local disk file system, the NFS file system, and the networking stack a second time. That is *many* subsystem maintainers who have to get involved.

In addition, the lock classification system is not documented at all, so now you also need someone who understands the lockdep code. And since some of these classifications involve transient objects, and lockdep doesn't have a way of dealing with transient locks, and has a hard compile time limit of the number of locks that it supports, to expect a subsystem maintainer to figure out all of the interactions, plus figure out lockdep, and work around lockdep's limitations seems.... not realistic.

(By the way, I've tried reading the crosslock and crossrelease documentation --- and I'm lost. Sorry, I'm just not smart enough to understand how it works, at least not from reading the documentation that was in the patch series. And honestly, I don't care. All I do need is some practical instructions for how to "classify locks properly", and how this interacts with lockdep's limitations.)

Quoted 7 lines

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

Ingo's argument is that (1) is not going to be happening any time soon, and in the meantime, code which is turned off will bitrot.

Given that once Lockdep reports a locking violation, it doesn't report any more lockdep violations, if there are a large number of false positives, people will not want to turn on cross-release, since it will report the false positive and then turn itself off, so it won't report anything useful. So if no one turns it on because of the false positives, how does the bitrot problem get resolved?

And if the answer is that some small number of lockdep experts will be trying to figure out how to do (1) in a tractable way, then Ingo has argued it can be handled via an out-of-tree patch.

Quoted 1 line

(3) The 3rd way: To invalidate waiters making trouble.

Hmm, can we make cross-release and cross-lock off by default on a per lock basis? With a well documented to enable it? I'm still not sure how this works given the cross-subsystem problem, though.

So if networking enables it because there are no problems with their TCP-only test, and then it blows up when someone is doing NBD or NFS testing, what's the recourse? The file system developer submitting a patch against the networking subsystem to turn off the lockdep tracking on that particular lock because it's causing pain for the file system developer? I can see that potentially causing all sorts of inter-subsystem conflicts.

Quoted 6 lines

Talking about what Ingo said in the commit msg.. I want to ask him back, if he did it with no false positives at the moment merging it in 2006, without using (2) or (3) method. I bet he know what it means.. And classifying locks/waiters correctly is not something uglifying code but a way to document code better. I've felt ill at ease because of the unnatural and forced explanation.

So I think this is the big difference is that potential for cross-subsystem false positives is dramatically higher than with cross-release compared with the traditional lockdep. And I'm not sure there is a clean solution --- how do you "cleanly classify" locks when in some cases each object's locks needs to be considered individual locks, and when that must not be done lest there is an explosion of the number of locks which lockdep needs to track (which is strictly limited due to memory and CPU overhead, as I understand things)? I haven't seen an explanation for how to solve this in a clean, general way --- and I strongly suspect it doesn't exist.

Regards,

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#18 lore raw
Quoted 5 lines · Theodore Ts'o
Quoted 2 lines · Byungchul Park

(1) The best way: To classify all waiters correctly.

It's really not all waiters, but all *locks*, no?

Thanks for your opinion. I will add my opinion on you.

I meant *waiters*. Locks are only a sub set of potential waiters, which actually cause deadlocks. Cross-release was designed to consider the super set including all general waiters such as typical locks, wait_for_completion(), and lock_page() and so on..

Quoted 12 lines
Quoted 8 lines

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

The problem is that it's not one subsystem, but *many*. And it's the interactions between the subsystems.

Quoted 6 lines

Consider the example I gave of a network block device, on which a local disk file system is mounted, which is then exported over NFS. So we have the Networking (TCP) stack involved, the NBD device driver, the local disk file system, the NFS file system, and the networking stack a second time. That is *many* subsystem maintainers who have to get involved.

I admit that it's not simple one to solve..

Quoted 8 lines

In addition, the lock classification system is not documented at all, so now you also need someone who understands the lockdep code. And since some of these classifications involve transient objects, and lockdep doesn't have a way of dealing with transient locks, and has a hard compile time limit of the number of locks that it supports, to expect a subsystem maintainer to figure out all of the interactions, plus figure out lockdep, and work around lockdep's limitations seems.... not realistic.

I have to think it more to find out how to solve it simply enough to be acceptable. The only solution I come up with for now is too complex.

Quoted 4 lines

(By the way, I've tried reading the crosslock and crossrelease documentation --- and I'm lost. Sorry, I'm just not smart enough to understand how it works, at least not from reading the documentation that was in the patch series. And honestly, I don't care. All I do

I am sorry for that. My english is too bad.. I can explain whatever you wonder if you ask me.

Quoted 2 lines

need is some practical instructions for how to "classify locks properly", and how this interacts with lockdep's limitations.)

I see what you consider. As you know, it's not something that I can solve right away. That's why I suggested (2) or (3)..

Quoted 7 lines
Quoted 7 lines

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

^
should be (3)

Quoted 2 lines

Ingo's argument is that (1) is not going to be happening any time soon, and in the meantime, code which is turned off will bitrot.

The root cause of the problem is that locks, generally speaking, waiters are roughly classified. IOW, having the new code with a better classification is worth, even it would be done later.

Quoted 6 lines

Given that once Lockdep reports a locking violation, it doesn't report any more lockdep violations, if there are a large number of false positives, people will not want to turn on cross-release, since it will report the false positive and then turn itself off, so it won't report anything useful. So if no one turns it on because of the false positives, how does the bitrot problem get resolved?

The problems come from wrong classification. Waiters either classfied well or invalidated properly won't bitrot.

Quoted 8 lines

And if the answer is that some small number of lockdep experts will be trying to figure out how to do (1) in a tractable way, then Ingo has argued it can be handled via an out-of-tree patch.

Quoted 1 line

(3) The 3rd way: To invalidate waiters making trouble.

Hmm, can we make cross-release and cross-lock off by default on a per lock basis? With a well documented to enable it? I'm still not sure

Yes. More precisely speaking, we can make cross-release check off on a per waiter basis, for example, by using init_completion_nomap() or its family which I can provide if needed, leaving other traditional lockdep checking *unchanged*. For that issue we talked about, we could use it in submit_bio_wait() to invalidate the checking for the waiter.

Quoted 1 line

how this works given the cross-subsystem problem, though.

It works because the invalidation make lockdep not generate the link between a set of fs locks on a layer and another set on another layer.

Quoted 7 lines

So if networking enables it because there are no problems with their TCP-only test, and then it blows up when someone is doing NBD or NFS testing, what's the recourse? The file system developer submitting a patch against the networking subsystem to turn off the lockdep tracking on that particular lock because it's causing pain for the file system developer? I can see that potentially causing all sorts of inter-subsystem conflicts.

If it can never be solved anyway, we can invalidate the waiter. What I want to say is that it's better than nothing, since cross-release would work and give the benefit in most cases, except that complicated case.

Quoted 7 lines
Quoted 6 lines

Talking about what Ingo said in the commit msg.. I want to ask him back, if he did it with no false positives at the moment merging it in 2006, without using (2) or (3) method. I bet he know what it means.. And classifying locks/waiters correctly is not something uglifying code but a way to document code better. I've felt ill at ease because of the unnatural and forced explanation.

Quoted 10 lines

So I think this is the big difference is that potential for cross-subsystem false positives is dramatically higher than with cross-release compared with the traditional lockdep. And I'm not sure there is a clean solution --- how do you "cleanly classify" locks when in some cases each object's locks needs to be considered individual locks, and when that must not be done lest there is an explosion of the number of locks which lockdep needs to track (which is strictly limited due to memory and CPU overhead, as I understand things)? I haven't seen an explanation for how to solve this in a clean, general way --- and I strongly suspect it doesn't exist.

I think this is the main point you want to point out anyway. Couldn't we why, if we try in one way or another?

For example, we can introduce the concept of group so classes in each group can be distinguished from another, of course, there might also be many things to discuss though.

--
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#19 lore raw
Quoted 5 lines · Byungchul Park
Quoted 4 lines · Theodore Ts'o
Quoted 2 lines · Byungchul Park

(1) The best way: To classify all waiters correctly.

Quoted 8 lines
Quoted 1 line

It's really not all waiters, but all *locks*, no?

Thanks for your opinion. I will add my opinion on you.

I meant *waiters*. Locks are only a sub set of potential waiters, which actually cause deadlocks. Cross-release was designed to consider the super set including all general waiters such as typical locks, wait_for_completion(), and lock_page() and so on..

I think this is a terminology problem. To me (and, I suspect Ted), a waiter is a subject of a verb while a lock is an object. So Ted is asking whether we have to classify the users, while I think you're saying we have extra objects to classify.

I'd be comfortable continuing to refer to completions as locks. We could try to come up with a new object name like waitpoints though?

Quoted 11 lines
Quoted 8 lines

In addition, the lock classification system is not documented at all, so now you also need someone who understands the lockdep code. And since some of these classifications involve transient objects, and lockdep doesn't have a way of dealing with transient locks, and has a hard compile time limit of the number of locks that it supports, to expect a subsystem maintainer to figure out all of the interactions, plus figure out lockdep, and work around lockdep's limitations seems.... not realistic.

I have to think it more to find out how to solve it simply enough to be acceptable. The only solution I come up with for now is too complex.

I want to amplify Ted's point here. How to use the existing lockdep functionality is undocumented. And that's not your fault. We have Documentation/locking/lockdep-design.txt which I'm sure is great for someone who's willing to invest a week understanding it, but we need a "here's how to use it" guide.

Quoted 9 lines
Quoted 6 lines

Given that once Lockdep reports a locking violation, it doesn't report any more lockdep violations, if there are a large number of false positives, people will not want to turn on cross-release, since it will report the false positive and then turn itself off, so it won't report anything useful. So if no one turns it on because of the false positives, how does the bitrot problem get resolved?

The problems come from wrong classification. Waiters either classfied well or invalidated properly won't bitrot.

I disagree here. As Ted says, it's the interactions between the subsystems that leads to problems. Everything's goig to work great until somebody does something in a way that's never been tried before.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#20 lore raw
Quoted 5 lines · Matthew Wilcox

I think this is a terminology problem. To me (and, I suspect Ted), a waiter is a subject of a verb while a lock is an object. So Ted is asking whether we have to classify the users, while I think you're saying we have extra objects to classify.

Exactly, the classification is applied when the {lock, mutex, completion} object is initialized. Not currently at the individual call points to mutex_lock(), wait_for_completion(), down_write(), etc.

Quoted 6 lines
Quoted 2 lines

The problems come from wrong classification. Waiters either classfied well or invalidated properly won't bitrot.

I disagree here. As Ted says, it's the interactions between the subsystems that leads to problems. Everything's goig to work great until somebody does something in a way that's never been tried before.

The question what is classified *well* mean? At the extreme, we could put the locks for every single TCP connection into their own lockdep class. But that would blow the limits in terms of the number of locks out of the water super-quickly --- and it would destroy the ability for lockdep to learn what the proper locking order should be. Yet given Lockdep's current implementation, the only way to guarantee that there won't be any interactions between subsystems that cause false positives would be to categorizes locks for each TCP connection into their own class.

So this is why I get a little annoyed when you say, "it's just a matter of classification". NO IT IS NOT. We can not possibly classify things "correctly" to completely limit false positives without completely destroying lockdep's scalability as it is currently designed. Byungchul, you don't acknowledge this, and it makes the "just classify everything" argument completely suspect as a result.

As far as the "just invalidate the waiter", the problem is that it requires source level changes to invalidate the waiter, and for different use cases, we will need to validate different waiters. For example, in the example I gave, we would have to invalidate *all* TCP waiters/locks in order to prevent false positives. But that makes the lockdep useless for all TCP locks. What's the solution? I claim that until lockdep is fundamentally fixed, there is no way to eliminate *all* false positives without invalidating *all* cross-release/cross-locks --- in which case you might as well leave the cross-release patches as an out of tree patch.

So to claim that we can somehow fix the problem by making source-level changes outside of lockdep, by "properly classifying" or "properly invalidating" all locks, just doesn't make sense.

The only way it can work is to either dump it on the reposibility of the people debugging lockdep reports to make source level changes to other subsystems which they aren't the maintainers of to suppress false positives that arise due to how the subsystems are being used together in their particular configuration ---- or you can try to claim that there is an "acceptable level" of false positives with which we can live with forever, and which can not be fixed by "proper classifying" the locks.

Or you can try to make lockdep scalable enough that if we could put every single lock for every single object into its own lock class (e.g., each lock for every single TCP connection gets its own lock class) which is after all the only way we can "properly classify everything") and still let lockdep be useful.

If you think that is doable, why don't you work on that, and once that is done, maybe cross-locks lockdep will be considered more acceptable for mainstream?

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#21 lore raw
Quoted 8 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox
Quoted 2 lines

The problems come from wrong classification. Waiters either classfied well or invalidated properly won't bitrot.

I disagree here. As Ted says, it's the interactions between the subsystems that leads to problems. Everything's goig to work great until somebody does something in a way that's never been tried before.

Quoted 9 lines

The question what is classified *well* mean? At the extreme, we could put the locks for every single TCP connection into their own lockdep class. But that would blow the limits in terms of the number of locks out of the water super-quickly --- and it would destroy the ability for lockdep to learn what the proper locking order should be. Yet given Lockdep's current implementation, the only way to guarantee that there won't be any interactions between subsystems that cause false positives would be to categorizes locks for each TCP connection into their own class.

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#22 lore raw
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

The point I was trying to drive home is that "all we have to do is just classify everything well or just invalidate the right lock objects" is a massive understatement of the complexity level of what would be required, or the number of locks/completion handlers that would have to be blacklisted.

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#23 lore raw
Quoted 8 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

Quoted 7 lines

I was exagerrating a bit for effect, I admit. (but only a bit).

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

So "all you have to do is classify the locks 'properly'" is much like the apocrophal, "all you have to do is bell the cat"[1]. Or like the saying, "colonizing the stars is *easy*; all you have to do is figure out faster than light travel."

[1] https://en.wikipedia.org/wiki/Belling_the_cat

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#24 lore raw
Quoted 10 lines · Theodore Ts'o
Quoted 9 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

I feel like there's been rather too much of that recently. Can we stick to facts as far as possible, please?

Quoted 6 lines
Quoted 5 lines

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

Quoted 6 lines

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem, local filesystem sends write to block device, block device sends a packet to a socket which takes that socket lock.

I don't think we need to be as drastic as giving each socket its own lock class to solve this. All NFS sockets can be in lock class A; all NBD sockets can be in lock class B; all user sockets can be in lock class C; etc.

Quoted 4 lines

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

Yes! I'd love to have a lockdep expert weigh in here. I believe it's legitimate to change a lock's class after it's been used, essentially destroying it and reinitialising it. If not, it should be because it's a reasonable design for an object to need different lock classes for different phases of its existance.

Quoted 4 lines

So "all you have to do is classify the locks 'properly'" is much like the apocrophal, "all you have to do is bell the cat"[1]. Or like the saying, "colonizing the stars is *easy*; all you have to do is figure out faster than light travel."

This is only computer programming, not rocket surgery :-)

#25 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 6 lines

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem, local filesystem sends write to block device, block device sends a packet to a socket which takes that socket lock.

It's not just the socket lock, but any of the locks/mutexes/"waiters" that might be taken in the TCP code path and below, including in the NIC driver.

Quoted 4 lines

I don't think we need to be as drastic as giving each socket its own lock class to solve this. All NFS sockets can be in lock class A; all NBD sockets can be in lock class B; all user sockets can be in lock class C; etc.

But how do you know which of the locks taken in the networking stack are for the NBD versus the NFS sockets? What manner of horrific abstraction violation is going to pass that information all the way down to all of the locks that might be taken at the socket layer and below?

How is this "proper clasification" supposed to happen? It's the repeated handwaving which claims this is easy which is rather frustrating. The simple thing is to use a unique ID which is bumped for each struct sock, each struct super, struct block_device, struct request_queue, struct bdi, etc, but that runs into lockdep scalability issues.

Anything else means that you have to somehow pass down through the layers so that, in the general case, the socket knows that it is "an NFS socket" versus "an NBD socket" --- and remember, if there is any kind of completion handling done in the NIC driver, it's going to have to passed down well below the TCP layer all the way down to the network device drivers. Or is the plan to do this add a bit ad hoc of plumbing for each false positive which cross-release lockdep failures are reported?

Quoted 10 lines
Quoted 4 lines

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

Yes! I'd love to have a lockdep expert weigh in here. I believe it's legitimate to change a lock's class after it's been used, essentially destroying it and reinitialising it. If not, it should be because it's a reasonable design for an object to need different lock classes for different phases of its existance.

We just also need to be destroy a lock class after the transient object has been deleted. This is especially true for file system testing, since we are constantly mounting and unmounting file systems, and creating and destroying loop devices, potentially hundreds or thousands of times during a test run. So if we have to create a unique lock class for "proper classification" each time a file system is mounted, or loop device or device-mapper device (dm-error, etc.) is created, we'll run into lockdep scalability issues really quickly.

So this is yet another example where the handwaving, "all you have to do is proper classification" just doesn't work.

Quoted 6 lines
Quoted 4 lines

So "all you have to do is classify the locks 'properly'" is much like the apocrophal, "all you have to do is bell the cat"[1]. Or like the saying, "colonizing the stars is *easy*; all you have to do is figure out faster than light travel."

This is only computer programming, not rocket surgery :-)

Given the current state of the lockdep technology, merging cross-lock certainly feels like requiring the use of sledgehammers to do rocket surgery in order to avoid false positives --- sorry, "proper classification".

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#26 lore raw
Quoted 28 lines · Theodore Ts'o
Quoted 11 lines · Matthew Wilcox
Quoted 6 lines

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem, local filesystem sends write to block device, block device sends a packet to a socket which takes that socket lock.

It's not just the socket lock, but any of the locks/mutexes/"waiters" that might be taken in the TCP code path and below, including in the NIC driver.

Quoted 4 lines

I don't think we need to be as drastic as giving each socket its own lock class to solve this. All NFS sockets can be in lock class A; all NBD sockets can be in lock class B; all user sockets can be in lock class C; etc.

But how do you know which of the locks taken in the networking stack are for the NBD versus the NFS sockets? What manner of horrific abstraction violation is going to pass that information all the way down to all of the locks that might be taken at the socket layer and below?

Quoted 6 lines

How is this "proper clasification" supposed to happen? It's the repeated handwaving which claims this is easy which is rather frustrating. The simple thing is to use a unique ID which is bumped for each struct sock, each struct super, struct block_device, struct request_queue, struct bdi, etc, but that runs into lockdep scalability issues.

This is what I mentioned with group ID in an example for you before. To do that, the most important thing is to prevent running into lockdep scalability.

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#27 lore raw
In reply to Matthew Wilcox
Quoted 43 lines · Matthew Wilcox
Quoted 10 lines · Theodore Ts'o
Quoted 9 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

I feel like there's been rather too much of that recently. Can we stick to facts as far as possible, please?

Quoted 6 lines
Quoted 5 lines

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

Quoted 6 lines

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem, local filesystem sends write to block device, block device sends a packet to a socket which takes that socket lock.

I don't think we need to be as drastic as giving each socket its own lock class to solve this. All NFS sockets can be in lock class A; all NBD sockets can be in lock class B; all user sockets can be in lock class C; etc.

Quoted 4 lines

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

Quoted 5 lines

Yes! I'd love to have a lockdep expert weigh in here. I believe it's legitimate to change a lock's class after it's been used, essentially destroying it and reinitialising it. If not, it should be because it's a reasonable design for an object to need different lock classes for different phases of its existance.

I also think it should be done ultimately. And I think it's very much hard since it requires to change the dependency graph of lockdep but anyway possible. It's up to lockdep maintainer's will though..

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Reviewed-by Rich Johnston
#28 lore raw
Quoted 13 lines · Byungchul Park
Quoted 11 lines · Matthew Wilcox
Quoted 4 lines · Theodore Ts'o

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

Yes! I'd love to have a lockdep expert weigh in here. I believe it's legitimate to change a lock's class after it's been used, essentially destroying it and reinitialising it. If not, it should be because it's a reasonable design for an object to need different lock classes for different phases of its existance.

Quoted 3 lines

I also think it should be done ultimately. And I think it's very much hard since it requires to change the dependency graph of lockdep but anyway possible. It's up to lockdep maintainer's will though..

We used to do this in XFS to work around the fact that the memory reclaim context "locks" were too stupid to understand that an object referenced and locked above memory allocation could not be accessed below in memory reclaim because memory reclaim only accesses /unreferenced objects/. We played whack-a-mole with lockdep for years to get most of the false positives sorted out.

Hence for a long time we had to re-initialise the lock context for the XFS inode iolock in ->evict_inode() so we could lock it for reclaim processing. Eventually we ended up completely reworking the inode reclaim locking in XFS primarily to get rid of all the nasty lockdep hacks we had strewn throughout the code. It was ~2012 we got rid of the last inode re-init code, IIRC. Yeah:

commit 4f59af758f9092bc7b266ca919ce6067170e5172

Author: Christoph Hellwig <[email protected]>
Date:   Wed Jul 4 11:13:33 2012 -0400

xfs: remove iolock lock classes

Now that we never take the iolock during inode reclaim we don't need to play games with lock classes.

We still have problems with lockdep false positives w.r.t. memory allocation contexts, mainly with code that can be called from both above and below memory allocation contexts. We've finally got __GFP_NOLOCKDEP to be able to annotate memory allocation points within such code paths, but that doesn't help with locks....

Byungchul, lockdep has a long, long history of having sharp edges and being very unfriendly to developers. We've all been scarred by lockdep at one time or another and so there's a fair bit of resistance to repeating past mistakes and allowing lockdep to inflict more scars on us....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#29 lore raw
Quoted 51 lines · Dave Chinner
Quoted 13 lines · Byungchul Park
Quoted 11 lines · Matthew Wilcox
Quoted 4 lines · Theodore Ts'o

Also, what to do with TCP connections which are created in userspace (with some authentication exchanges happening in userspace), and then passed into kernel space for use in kernel space, is an interesting question.

Yes! I'd love to have a lockdep expert weigh in here. I believe it's legitimate to change a lock's class after it's been used, essentially destroying it and reinitialising it. If not, it should be because it's a reasonable design for an object to need different lock classes for different phases of its existance.

Quoted 3 lines

I also think it should be done ultimately. And I think it's very much hard since it requires to change the dependency graph of lockdep but anyway possible. It's up to lockdep maintainer's will though..

We used to do this in XFS to work around the fact that the memory reclaim context "locks" were too stupid to understand that an object referenced and locked above memory allocation could not be accessed below in memory reclaim because memory reclaim only accesses /unreferenced objects/. We played whack-a-mole with lockdep for years to get most of the false positives sorted out.

Hence for a long time we had to re-initialise the lock context for the XFS inode iolock in ->evict_inode() so we could lock it for reclaim processing. Eventually we ended up completely reworking the inode reclaim locking in XFS primarily to get rid of all the nasty lockdep hacks we had strewn throughout the code. It was ~2012 we got rid of the last inode re-init code, IIRC. Yeah:

commit 4f59af758f9092bc7b266ca919ce6067170e5172

Author: Christoph Hellwig <[email protected]>
Date:   Wed Jul 4 11:13:33 2012 -0400

xfs: remove iolock lock classes

Now that we never take the iolock during inode reclaim we don't need to play games with lock classes.

We still have problems with lockdep false positives w.r.t. memory allocation contexts, mainly with code that can be called from both above and below memory allocation contexts. We've finally got __GFP_NOLOCKDEP to be able to annotate memory allocation points within such code paths, but that doesn't help with locks....

Quoted 5 lines

Byungchul, lockdep has a long, long history of having sharp edges and being very unfriendly to developers. We've all been scarred by lockdep at one time or another and so there's a fair bit of resistance to repeating past mistakes and allowing lockdep to inflict more scars on us....

As I understand what you suffered from.. I don't really want to force it forward strongly.

So far, all problems have been handled by myself including the final one e.i. the completion in submit_bio_wait() with the invalidation if it's allowed. But yes, who knows the future? In the future, that terrible thing you mentioned might or might not happen because of cross-release.

I just felt like someone was misunderstanding what the problem came from, what the problem was, how we could avoid it, why cross-release should be removed and so on..

I believe the 3 ways I suggested can help, but I don't want to strongly insist if all of you don't think so.

Thanks a lot anyway for your opinion.

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#30 lore raw
In reply to Matthew Wilcox
Quoted 28 lines · Matthew Wilcox
Quoted 10 lines · Theodore Ts'o
Quoted 9 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

I feel like there's been rather too much of that recently. Can we stick to facts as far as possible, please?

Quoted 6 lines
Quoted 5 lines

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

Quoted 6 lines

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

Quoted 2 lines

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem,

I'm confused, what lock does Ted think the NFS server is holding over NFS processing?

--b.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#31 lore raw
Quoted 29 lines · bfields
Quoted 28 lines · Matthew Wilcox
Quoted 10 lines · Theodore Ts'o
Quoted 9 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

I feel like there's been rather too much of that recently. Can we stick to facts as far as possible, please?

Quoted 12 lines
Quoted 5 lines

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

Clarification: all TCP connections that are used by kernel code would need to be in their own separate lock class. All TCP connections used only by userspace could be in their own shared lock class. You can't use a one lock class for all kernel-used TCP connections, because of the Network Block Device mounted on a local file system which is then exported via NFS and squirted out yet another TCP connection problem.

Quoted 5 lines
Quoted 2 lines

So the false positive you're concerned about is write-comes-in-over-NFS (with socket lock held), NFS sends a write request to local filesystem,

I'm confused, what lock does Ted think the NFS server is holding over NFS processing?

Sorry, I meant "over RPC processing".

I'll confess to no understanding of socket locking. The server RPC code doesn't take any itself except in a couple places on setup and tear down of a connection. We wouldn't actually want any exclusive per-connection lock held across RPC processing because we want to be able to handle multiple concurrent RPCs per connection.

We do need a little locking just to make sure multiple server threads replying to the same client don't accidentally corrupt their replies by interleaving. But even there we're using our own lock, held only while transmitting the reply (after all the work's done and reply encoded).

--b.

#32 lore raw
In reply to Theodore Ts'o
Quoted 10 lines · Theodore Ts'o
Quoted 6 lines · Matthew Wilcox

I'm not sure I agree with this part. What if we add a new TCP lock class for connections which are used for filesystems/network block devices/...? Yes, it'll be up to each user to set the lockdep classification correctly, but that's a relatively small number of places to add annotations, and I don't see why it wouldn't work.

I was exagerrating a bit for effect, I admit. (but only a bit).

Quoted 8 lines

It can probably be for all TCP connections that are used by kernel code (as opposed to userspace-only TCP connections). But it would probably have to be each and every device-mapper instance, each and every block device, each and every mounted file system, each and every bdi object, etc.

The point I was trying to drive home is that "all we have to do is just classify everything well or just invalidate the right lock

Just to be sure, we don't have to invalidate lock objects at all but a problematic waiter only.

Quoted 6 lines

objects" is a massive understatement of the complexity level of what would be required, or the number of locks/completion handlers that would have to be blacklisted.

- Ted

Signature · 2 lines
Thanks,
Byungchul
#33 lore raw
Quoted 5 lines · Byungchul Park
Quoted 2 lines

The point I was trying to drive home is that "all we have to do is just classify everything well or just invalidate the right lock

Just to be sure, we don't have to invalidate lock objects at all but a problematic waiter only.

So essentially you are proposing that we have to play "whack-a-mole" as we find false positives, and where we may have to put in ad-hoc plumbing to only invalidate "a problematic waiter" when it's problematic --- or to entirely suppress the problematic waiter altogether. And in that case, a file system developer might be forced to invalidate a lock/"waiter"/"completion" in another subsystem.

I will also remind you that doing this will trigger a checkpatch.pl *error*:

ERROR("LOCKDEP", "lockdep_no_validate class is reserved for device->mutex.\n" . $herecurr);

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#34 lore raw
Quoted 11 lines · Theodore Ts'o
Quoted 5 lines · Byungchul Park
Quoted 2 lines

The point I was trying to drive home is that "all we have to do is just classify everything well or just invalidate the right lock

Just to be sure, we don't have to invalidate lock objects at all but a problematic waiter only.

So essentially you are proposing that we have to play "whack-a-mole" as we find false positives, and where we may have to put in ad-hoc plumbing to only invalidate "a problematic waiter" when it's problematic --- or to entirely suppress the problematic waiter

If we have too many problematic completions(waiters) to handle it, then I agree with you. But so far, only one exits and it seems able to be handled even in the future on my own.

Or if you believe that we have a lot of those kind of completions making trouble so we cannot handle it, the (4) by Amir would work, no? I'm asking because I'm really curious about your opinion..

Quoted 2 lines

altogether. And in that case, a file system developer might be forced to invalidate a lock/"waiter"/"completion" in another subsystem.

As I said, with regard to the invalidation, we don't have to consider locks at all. It's enough to invalidate the waiter only.

Quoted 2 lines

I will also remind you that doing this will trigger a checkpatch.pl *error*:

This is what we decided. And I think the decision is reasonable for original lockdep. But I wonder if we should apply the same decision on waiters. I don't insist but just wonder.

Quoted 4 lines

ERROR("LOCKDEP", "lockdep_no_validate class is reserved for device->mutex.\n" . $herecurr);

- Ted

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#35 lore raw
Quoted 27 lines · Byungchul Park
Quoted 11 lines · Theodore Ts'o
Quoted 5 lines · Byungchul Park
Quoted 2 lines

The point I was trying to drive home is that "all we have to do is just classify everything well or just invalidate the right lock

Just to be sure, we don't have to invalidate lock objects at all but a problematic waiter only.

So essentially you are proposing that we have to play "whack-a-mole" as we find false positives, and where we may have to put in ad-hoc plumbing to only invalidate "a problematic waiter" when it's problematic --- or to entirely suppress the problematic waiter

If we have too many problematic completions(waiters) to handle it, then I agree with you. But so far, only one exits and it seems able to be handled even in the future on my own.

Or if you believe that we have a lot of those kind of completions making trouble so we cannot handle it, the (4) by Amir would work, no? I'm asking because I'm really curious about your opinion..

Quoted 2 lines

altogether.  And in that case, a file system developer might be forced to invalidate a lock/"waiter"/"completion" in another subsystem.

As I said, with regard to the invalidation, we don't have to consider locks at all. It's enough to invalidate the waiter only.

Quoted 6 lines
Quoted 2 lines

I will also remind you that doing this will trigger a checkpatch.pl *error*:

This is what we decided. And I think the decision is reasonable for original lockdep. But I wonder if we should apply the same decision on waiters. I don't insist but just wonder.

What if we adopt the (4) in which waiters are validated one by one and no explicit invalidation is involved?

Quoted 5 lines
Quoted 4 lines

ERROR("LOCKDEP", "lockdep_no_validate class is reserved for device->mutex.\n" . $herecurr);

                                      - Ted

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#36 lore raw
In reply to Theodore Ts'o
Quoted 16 lines · Theodore Ts'o
Quoted 4 lines · Matthew Wilcox

I disagree here. As Ted says, it's the interactions between the subsystems that leads to problems. Everything's goig to work great until somebody does something in a way that's never been tried before.

The question what is classified *well* mean? At the extreme, we could put the locks for every single TCP connection into their own lockdep class. But that would blow the limits in terms of the number of locks out of the water super-quickly --- and it would destroy the ability for lockdep to learn what the proper locking order should be. Yet given Lockdep's current implementation, the only way to guarantee that there won't be any interactions between subsystems that cause false positives would be to categorizes locks for each TCP connection into their own class.

Quoted 4 lines

So this is why I get a little annoyed when you say, "it's just a matter of classification". NO IT IS NOT. We can not possibly classify things "correctly" to completely limit false positives without completely destroying lockdep's scalability as it is currently

You seem to admit that it can be solved by proper classification but say that it's *not realistic* because of the limitation of lockdep.

Right?

I've agreed with you for that point. I also think it's very hard to do it because of the lockdep design and the only way might be to fix lockdep fundamentally, that may be the one we should do ultimately.

Is it the best decision to keep it removed until lockdep get fixed fundamentally? If I believe it were, I would have kept quiet. But, I don't think so. Almost other users had already gotten benifit from it except the special case.

And it would be appriciated if you remind that I suggested 3 methods
+ 1 (by Amir) before for that reason.

I don't want to force it forward but just want the facts to be shared. I felt like I failed it because of the lack of explanation.

Quoted 2 lines

As far as the "just invalidate the waiter", the problem is that it requires source level changes to invalidate the waiter, and for

Or, no change is needed if we adopt the (4)th option (by Amir), in which we keep waiters invalidated by default and validate waiters explicitly only when it needs.

Quoted 3 lines

different use cases, we will need to validate different waiters. For example, in the example I gave, we would have to invalidate *all* TCP waiters/locks in order to prevent false positives. But that makes the

No. Only invalidating waiters is enough. For now, the waiter in submit_bio_wait() is the only one to invalidate.

Quoted 1 line

lockdep useless for all TCP locks. What's the solution? I claim that

Even if we invalidate waiters, TCP locks can still work with lockdep. Invalidating waiters *never* affect lockdep checking for typical locks at all.

Quoted 5 lines

The only way it can work is to either dump it on the reposibility of the people debugging lockdep reports to make source level changes to other subsystems which they aren't the maintainers of to suppress false positives that arise due to how the subsystems are being used together in their particular configuration ---- or you can try to

You seem to misunderstand it a lot.. The only thing we have to is to use init_completion_nomap() instead of init_completion() for the problematic completion object. So far, the completion in submit_bio_wait() has been the only one.

If you belive that we have a lot of problematic completions(waiters) so that we cannot handle it, the (4) by Amir can be an option.

Just to be sure, there were several false positives by cross-release. Something was due to confliction between manual acquire()s added before and automatic cross-release, both of which are for detecting deadlocks by a specific completion(waiter). Or, something was solved by classifying locks properly simply. And this case of submit_bio_wait() is the first case that we cannot classify locks simply and need to consider other options.

Signature · 8 lines
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#37 lore raw
In reply to Matthew Wilcox
Quoted 15 lines · Matthew Wilcox
Quoted 5 lines · Byungchul Park
Quoted 4 lines · Theodore Ts'o
Quoted 2 lines · Byungchul Park

(1) The best way: To classify all waiters correctly.

Quoted 8 lines
Quoted 1 line

It's really not all waiters, but all *locks*, no?

Thanks for your opinion. I will add my opinion on you.

I meant *waiters*. Locks are only a sub set of potential waiters, which actually cause deadlocks. Cross-release was designed to consider the super set including all general waiters such as typical locks, wait_for_completion(), and lock_page() and so on..

Quoted 7 lines

I think this is a terminology problem. To me (and, I suspect Ted), a waiter is a subject of a verb while a lock is an object. So Ted is asking whether we have to classify the users, while I think you're saying we have extra objects to classify.

I'd be comfortable continuing to refer to completions as locks. We could try to come up with a new object name like waitpoints though?

Right. Then "event" should be used as an object name than "waiter".

Quoted 4 lines
Quoted 2 lines

The problems come from wrong classification. Waiters either classfied well or invalidated properly won't bitrot.

I disagree here. As Ted says, it's the interactions between the

As you know, the classification is something already considering the interactions between the subsystems and classified. But, yes. That is just what we have to do untimately but not what we can do right away. That's why I suggested all 3 ways + 1 way (by Amir).

Quoted 2 lines

subsystems that leads to problems. Everything's goig to work great until somebody does something in a way that's never been tried before.

Yes. Everything has worked great so far, since the classification by now is done well enough at least to avoid such problems, not perfect though. IMO, the classification does not have to be perfect but needs to be good enough to work.

--
Thanks,
Byungchul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#38 lore raw
In reply to Byungchul Park
Quoted 59 lines · Byungchul Park
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

I admit the cross-release feature had introduced several false positives about 4 times(?), maybe. And I suggested roughly 3 ways to solve it. I should have explained each in more detail. The lack might have led some to misunderstand.

(1) The best way: To classify all waiters correctly.

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

(3) The 3rd way: To invalidate waiters making trouble.

Of course, this is not the best. Now that you have already spent a lot of time to fix original lockdep's problems since lockdep was introduced in 2006, we don't need to use this way for typical locks except a few special cases. Lockdep is fairly robust by now.

Quoted 5 lines

And I understand you don't want to spend more time to fix additional problems again. Now that the situation is different from the time, 2006, it's not too bad to use this way to handle the issues.

Purely logically, aren't you missing a 4th option:

(4) The 4th way: To validate specific waiters.

Is it not an option for a subsystem to opt-in for cross-release validation of specific locks/waiters? This may be a much preferred route for cross- release. I remember seeing a post from a graphic driver developer that found cross-release useful for finding bugs in his code.

For example, many waiters in kernel can be waiting for userspace code, so does that mean the cross-release is going to free the world from userspace deadlocks as well?? Possibly I am missing something.

In any way, it seem logical to me that some waiters should particpate in lock chain dependencies, while other waiters should break the chain to avoid false positives and to avoid protecting against user configurable deadlocks (like loop mount over file inside the loop mounted fs). And if you agree that this logic claim is correct, than surely, an inclusive approach is the best way forward.

Cheers,
Amir.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#39 lore raw
Quoted 66 lines · Amir Goldstein
Quoted 59 lines · Byungchul Park
Quoted 24 lines · Byungchul Park

Lockdep works, based on the following:

(1) Classifying locks properly
(2) Checking relationship between the classes

If (1) is not good or (2) is not good, then we might get false positives.

For (1), we don't have to classify locks 100% properly but need as enough as lockdep works.

For (2), we should have a mechanism w/o
logical defects.

Cross-release added an additional capacity to
(2) and requires (1) to get more precisely classified.

Since the current classification level is too low for cross-release to work, false positives are being reported frequently with enabling cross-release. Yes. It's a obvious problem. It needs to be off by default until the classification is done by the level that cross-release requires.

Quoted 2 lines

But, the logic (2) is valid and logically true. Please keep the code, mechanism, and logic.

I admit the cross-release feature had introduced several false positives about 4 times(?), maybe. And I suggested roughly 3 ways to solve it. I should have explained each in more detail. The lack might have led some to misunderstand.

(1) The best way: To classify all waiters correctly.

Ultimately the problems should be solved in this way. But it takes a lot of time so it's not easy to use the way right away. And I need helps from experts of other sub-systems.

While talking about this way, I made a trouble.. I still believe that each sub-system expert knows how to solve dependency problems most, since each has own dependency rule, but it was not about responsibility. I've never wanted to charge someone else it but me.

(2) The 2nd way: To make cross-release off by default.

At the beginning, I proposed cross-release being off by default. Honestly, I was happy and did it when Ingo suggested it on by default once lockdep on. But I shouldn't have done that but kept it off by default. Cross-release can make some happy but some unhappy until problems go away through (1) or (2).

(3) The 3rd way: To invalidate waiters making trouble.

Of course, this is not the best. Now that you have already spent a lot of time to fix original lockdep's problems since lockdep was introduced in 2006, we don't need to use this way for typical locks except a few special cases. Lockdep is fairly robust by now.

Quoted 5 lines

And I understand you don't want to spend more time to fix additional problems again. Now that the situation is different from the time, 2006, it's not too bad to use this way to handle the issues.

Quoted 4 lines

Purely logically, aren't you missing a 4th option:

(4) The 4th way: To validate specific waiters.

Hello,

Thanks for your opinion. I will add my opinion on you.

Quoted 2 lines

Is it not an option for a subsystem to opt-in for cross-release validation of specific locks/waiters? This may be a much preferred route for cross-

Yes. I think it can be a good option.

I think we have to choose a better one between (3) and (4) depending on the following:

In case that there are few waiters making trouble, it would be better to choose (3).

In case that there are a lot of waiter making trouble, it would be better to chosse (4).

I think (3) is better for now because there's only one or two cases making us hard to handle it. However, if you don't agree, I also think (4) can be an available option.

Quoted 6 lines

release. I remember seeing a post from a graphic driver developer that found cross-release useful for finding bugs in his code.

For example, many waiters in kernel can be waiting for userspace code, so does that mean the cross-release is going to free the world from userspace deadlocks as well?? Possibly I am missing something.

I don't see what you are saying exactly.. but cross-release can be used if we know (a) the spot waiting for an event and (3) the other spot triggering the event. Please explain it more if I miss something.

Quoted 4 lines

In any way, it seem logical to me that some waiters should particpate in lock chain dependencies, while other waiters should break the chain to avoid false positives and to avoid protecting against user configurable deadlocks (like loop mount over file inside the loop mounted fs).

For example, when we had cross-release enabled, the following chain was built and false positives were produced:

link 1: ext4 spin lock class A (in a lower fs) ->
waiter class B (in submit_bio_wait())

link 2: waiter class B (in submit_bio_wait()) ->
ext4 spin lock class A (in an upper fs)

Even though conceptually it should have been "class A in lower fs != class A in upper fs", current code registers these two as class A.

So we need to correct the chain like, using (1):

link 1: ext4 spin lock class A (in a lower fs) ->
waiter class B (in submit_bio_wait())

link 2: waiter class B (in submit_bio_wait()) ->
ext4 spin lock class *C* (in an upper fs)

Or using (3) or (4):

no link (because waiter class B does not exist anymore)

Quoted 2 lines

And if you agree that this logic claim is correct, than surely, an inclusive approach is the best way forward.

I'm also curious about other opinions..

Quoted 1 line

Cheers,

2> Amir.

Quoted 1 line
Signature · 2 lines
Thanks,
Byungchul