Policy regarding linux-next only changes
Subsystems (2)
Hi Mark,
While testing lockdep changes on linux-next, I found there is a commit ca65ccfdc5b3 ("locking/lockdep: make lockdep_print_held_locks() always print if CONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.
I asked Boqun and as a maintainer of the file he's not aware of such changes being made. This commit is being added to linux-next via the TOMOYO security module tree; checking the merge it appears that none of the changes pulled in via the TOMOYO tree is related to LSM at all:
e07d10ed0f23 lib/Kconfig.debug: enable CONFIG_PREEMPT_RT for syzbot kernels.8f598d86dc9c net: add "struct dst_entry" debugging295e1a24166f net: update dev_put()/dev_hold() debuggingca65ccfdc5b3 locking/lockdep: make lockdep_print_held_locks() always print if CONFIG_DEBUG_AID_FOR_SYZBOT=y 83d2b1cd5d27 lib/Kconfig.debug: add CONFIG_DEBUG_AID_FOR_SYZBOT option
For obvious reasons I have concern about this practice; being a maintainer shouldn't mean that you're allowed to add random stuff to linux-next. This reminds me of another case last month
https://lore.kernel.org/all/[email protected]/ where the added stuff via TOMOYO tree breaks Rust build on linux-next. I think this practice is questionable and I would like a clarification on the policy of adding stuff that is not upstream material.
I recognize that there is a need to add commits that serve as testing and debugging purpose to linux-next, but I think any of these changes should be temporary in nature and have acknowledgement from maintainers of touched code. Perhaps these changes should also be in a separate topic branch which gets pulled from linux-next temporarily.
Best,
Gary
Quoted 1 line · Gary Guo
Hi Mark,
Thank you for raising this problem.
Quoted 6 lines
While testing lockdep changes on linux-next, I found there is a commit
ca65ccfdc5b3("locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.
Yes, these patches are for debugging difficult problems.
For example, ca65ccfdc5b3 has been a concern for almost 8 years without progress because developers show little interest for this change:
https://lkml.kernel.org/r/1535975097-19080-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
https://lkml.kernel.org/r/[email protected]
Quoted 7 lines
I asked Boqun and as a maintainer of the file he's not aware of such changes being made. This commit is being added to linux-next via the TOMOYO security module tree; checking the merge it appears that none of the changes pulled in via the TOMOYO tree is related to LSM at all:
e07d10ed0f23lib/Kconfig.debug: enableCONFIG_PREEMPT_RTfor syzbot kernels.
e07d10ed0f23 is for reproducing https://syzkaller.appspot.com/bug?extid=3f51ad7ac3ae57a6fdcc
in linux-next so that we can debug this problem in linux-next.git / test patches before sending to linux.git .
Quoted 2 lines
8f598d86dc9cnet: add "structdst_entry" debugging295e1a24166fnet: update dev_put()/dev_hold() debugging
8f598d86dc9c and 295e1a24166f are for debugging https://syzkaller.appspot.com/bug?extid=e2af46126e0644cbebdd
in linux-next because netdev people are hardly responding to patches
(e.g. https://lkml.kernel.org/r/[email protected] ) that will finally make these patches unnecessary.
Quoted 1 line
ca65ccfdc5b3locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y
Explained at top.
Quoted 1 line
83d2b1cd5d27lib/Kconfig.debug: addCONFIG_DEBUG_AID_FOR_SYZBOToption
This is also a long-remaining concern without progress because developers show little interest for this change:
(2019-12-16 09:59) [PATCH] kconfig: Add kernel config option for fuzz testing.
https://lkml.kernel.org/r/[email protected]
(2020-03-07 13:58) [PATCH v2] Add kernel config option for fuzz testing.
https://lkml.kernel.org/r/[email protected]
(2020-03-08 16:13) Re: [PATCH v2] Add kernel config option for fuzz testing.
https://lkml.kernel.org/r/CAHk-=wjCcCmQig8w8QEfyqyXACLzDc7b4TSW-KzAMzmS-QvJ+Q@mail.gmail.com
(2020-04-13 06:33) [PATCH v3] Add kernel config option for tweaking kernel behavior.
https://lkml.kernel.org/r/[email protected]
(2020-04-13 18:13) Re: [PATCH v3] Add kernel config option for tweaking kernel behavior.
https://lkml.kernel.org/r/CAHk-=wgbMi2+VBN0SCEw9GeoiWgui034AOBwbt_dW9tdCa3Nig@mail.gmail.com
(2020-04-16 00:47) Re: [PATCH v3] Add kernel config option for tweaking kernel behavior.
https://lkml.kernel.org/r/[email protected]Quoted 8 lines
For obvious reasons I have concern about this practice; being a maintainer shouldn't mean that you're allowed to add random stuff to linux-next. This reminds me of another case last month
https://lore.kernel.org/all/[email protected]/ where the added stuff via TOMOYO tree breaks Rust build on linux-next. I think this practice is questionable and I would like a clarification on the policy of adding stuff that is not upstream material.
If maintainers were more responding to patches that help debugging difficult problems / avoiding stupid behaviors, many of these patches would not have been staying as "linux-next only" patches.
Quoted 6 lines
I recognize that there is a need to add commits that serve as testing and debugging purpose to linux-next, but I think any of these changes should be temporary in nature and have acknowledgement from maintainers of touched code. Perhaps these changes should also be in a separate topic branch which gets pulled from linux-next temporarily.
I'm happy if we can assign a git tree and a ML that are intended for temporary patches. But please note that asking fuzzers to test a tree for hosting temporary patches is not a choice. We need to test on massive instances to reproduce problems. linux.git is the best tree for reproducing problems because of massive instances. But since we don't want to pollute linux.git with temporary patches, I'm using linux-next.git as the place for reproducing problems.
This practice is questionable, but just rejecting with "stop doing this practice" won't help. I need interests / assistance from developers / maintainers for avoid doing this practice.
Quoted 4 lines · Tetsuo Handa
If maintainers were more responding to patches that help debugging difficult problems / avoiding stupid behaviors, many of these patches would not have been staying as "linux-next only" patches.
Did you get an agreement from others (e.g. from Mark himself or those maintainers), or not?
As I mentioned to you last time in the thread that Gary linked, the standard linux-next rules do not allow for this. Of course, rules can bent (or changed) when there is a need, but it sounds to me like you are surprising others, even maintainers of related code, by unilaterally adding whatever patches you deem important, without prior agreement.
If you think something is really important, and there was disagreement (I don't know the context), then you should try to work with others to make them see why it is important, e.g. bring it to the Ksummit list, to LPC, escalate it, etc. But doing things unilaterally breaks the trust, especially when it is not the first time.
And why do you say fuzzers cannot test whatever tree you need? Can you please elaborate what the actual problem is? Maybe it can be solved in another way?
(Cc'ing Greg here too since I see from your threads that you discussed with him patches for fuzzing in the past several times.)
Cheers,
Miguel
Boqun Feng
maintainer[Cc Locking & Greg]
Quoted 12 lines · Tetsuo Handa
Quoted 1 line · Gary Guo
Hi Mark,
Thank you for raising this problem.
Quoted 6 lines
While testing lockdep changes on linux-next, I found there is a commit
ca65ccfdc5b3("locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.
Quoted 8 lines
Yes, these patches are for debugging difficult problems.
For example,
ca65ccfdc5b3has been a concern for almost 8 years without progress because developers show little interest for this change:https://lkml.kernel.org/r/1535975097-19080-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
https://lkml.kernel.org/r/[email protected]
This email was from 6 years ago, if that's the latest version, I'm not so sure you could say "developers show little interest for this change". Maybe you could provide more context about this that I missed? Also Greg left a comment there, you neither replied nor send a version to resolve it. Again I'm not sure if that could count as "developers show little interest for this change".
Quoted 62 lines
Quoted 7 lines
I asked Boqun and as a maintainer of the file he's not aware of such changes being made. This commit is being added to linux-next via the TOMOYO security module tree; checking the merge it appears that none of the changes pulled in via the TOMOYO tree is related to LSM at all:
e07d10ed0f23lib/Kconfig.debug: enableCONFIG_PREEMPT_RTfor syzbot kernels.
e07d10ed0f23is for reproducing https://syzkaller.appspot.com/bug?extid=3f51ad7ac3ae57a6fdcc
in linux-next so that we can debug this problem in linux-next.git / test patches before sending to linux.git .Quoted 2 lines
8f598d86dc9cnet: add "structdst_entry" debugging295e1a24166fnet: update dev_put()/dev_hold() debugging
8f598d86dc9cand295e1a24166fare for debugging https://syzkaller.appspot.com/bug?extid=e2af46126e0644cbebdd
in linux-next because netdev people are hardly responding to patches
(e.g. https://lkml.kernel.org/r/[email protected] ) that will finally make these patches unnecessary.Quoted 1 line
ca65ccfdc5b3locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=yExplained at top.
Quoted 1 line
83d2b1cd5d27lib/Kconfig.debug: addCONFIG_DEBUG_AID_FOR_SYZBOToptionThis is also a long-remaining concern without progress because developers show little interest for this change:
(2019-12-16 09:59) [PATCH] kconfig: Add kernel config option for fuzz testing. https://lkml.kernel.org/r/[email protected] (2020-03-07 13:58) [PATCH v2] Add kernel config option for fuzz testing. https://lkml.kernel.org/r/[email protected] (2020-03-08 16:13) Re: [PATCH v2] Add kernel config option for fuzz testing. https://lkml.kernel.org/r/CAHk-=wjCcCmQig8w8QEfyqyXACLzDc7b4TSW-KzAMzmS-QvJ+Q@mail.gmail.com (2020-04-13 06:33) [PATCH v3] Add kernel config option for tweaking kernel behavior. https://lkml.kernel.org/r/[email protected] (2020-04-13 18:13) Re: [PATCH v3] Add kernel config option for tweaking kernel behavior. https://lkml.kernel.org/r/CAHk-=wgbMi2+VBN0SCEw9GeoiWgui034AOBwbt_dW9tdCa3Nig@mail.gmail.com (2020-04-16 00:47) Re: [PATCH v3] Add kernel config option for tweaking kernel behavior. https://lkml.kernel.org/r/[email protected]Quoted 8 lines
For obvious reasons I have concern about this practice; being a maintainer shouldn't mean that you're allowed to add random stuff to linux-next. This reminds me of another case last month
https://lore.kernel.org/all/[email protected]/ where the added stuff via TOMOYO tree breaks Rust build on linux-next. I think this practice is questionable and I would like a clarification on the policy of adding stuff that is not upstream material.If maintainers were more responding to patches that help debugging difficult problems / avoiding stupid behaviors, many of these patches would not have been staying as "linux-next only" patches.
Quoted 6 lines
I recognize that there is a need to add commits that serve as testing and debugging purpose to linux-next, but I think any of these changes should be temporary in nature and have acknowledgement from maintainers of touched code. Perhaps these changes should also be in a separate topic branch which gets pulled from linux-next temporarily.
Quoted 3 lines
I'm happy if we can assign a git tree and a ML that are intended for temporary patches. But please note that asking fuzzers to test a tree for hosting temporary patches is not a choice. We need to test on massive instances to reproduce problems. linux.git is the
I don't understand why it's not an option, couldn't you just ask syzbot to apply the patches only for linux-next, or merge a tree but only for linux-next? The point is not just syzbot consumes linux-next, everyone does. Although I trust your good intents, but temporarily patches (especially when they are silently merged into linux-next) can undermine what linux-next is supposed to be.
By "silently" I mean I didn't see you gave a heads-up about doing this either, e.g. an email saying "hey I'm including these into the linux-next". Maybe I'm missing something here?
Regards,
Boqun
Quoted 7 lines
best tree for reproducing problems because of massive instances. But since we don't want to pollute linux.git with temporary patches, I'm using linux-next.git as the place for reproducing problems.
This practice is questionable, but just rejecting with "stop doing this practice" won't help. I need interests / assistance from developers / maintainers for avoid doing this practice.
Quoted 11 lines · Boqun Feng
Quoted 11 lines
Quoted 5 lines
While testing lockdep changes on linux-next, I found there is a commit
ca65ccfdc5b3("locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.Yes, these patches are for debugging difficult problems.
For example,
ca65ccfdc5b3has been a concern for almost 8 years without progress because developers show little interest for this change:
Quoted 7 lines
Quoted 3 lines
https://lkml.kernel.org/r/1535975097-19080-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
https://lkml.kernel.org/r/[email protected]This email was from 6 years ago, if that's the latest version, I'm not so sure you could say "developers show little interest for this change". Maybe you could provide more context about this that I missed?
I don't remember whether some discussions are done somewhere else. But judging from the fact that code remains unchanged, nobody else has attempted or succeeded to make this change.
Peter Zijlstra has commented at
https://lkml.kernel.org/r/[email protected]
that disabling
if (p->state == TASK_RUNNING && p != current) return;
check increases UAF potential, and suggested a kernel command line parameter for whether to disable this check.
At the same time, Peter has thought that the risk of UAF is quite low. As far as I know, syzbot did not hit UAF caused by disabling this check (commit ca65ccfdc5b3 ("locking/lockdep: make lockdep_print_held_locks() always print if CONFIG_DEBUG_AID_FOR_SYZBOT=y"). So, I guess that we can add a switch (I prefer a kernel config option over a kernel command line parameter) for this change.
Quoted 4 lines
Also Greg
left a comment there, you neither replied nor send a version to resolve it. Again I'm not sure if that could count as "developers show little interest for this change".
Linus does not like use of kernel config options, but I don't like use of kernel command line parameters
( https://lkml.kernel.org/r/[email protected] ).
(1) Adding one switch for one behavior (either a kernel config option or a kernel command line parameter) is an endless game of maintaining list of switches. Unless someone finds that a dedicated switch is useful for specific case, number of switches should remain minimized.
(2) Boot-time switches (i.e. kernel command line parameters) which are visible to userspace (e.g. /proc/cmdline, /proc/1/cmdline, sysfs and maybe more) are more difficult to maintain (i.e. follow the "never break the userspace" rule) than compile-time switches (i.e. kernel config options) because the kernel implementation might change in a way that makes boot-time switches unnecessary (e.g. switches that are intended for debugging will become unnecessary after some problem is fixed).
(3) Run-time switch increases size of executable files and cost for runtime evaluation. Also, assembly dump will include pointless code which is never executed due to run-time switch. Excluding should-not-be-executed-code from the beginning minimizes size and cost.
(4) Compile-time switch is easier to prove configurations than scanning very very long kernel command line which is likely already excluded from the console log of a crash report.
As a whole, I prefer compile-time and a global switch. If someone finds that run-time and/or individual switch is more useful, we can change to run-time and/or individual switch as needed. But no more response from Greg or Linus...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I recognize that there is a need to add commits that serve as testing and debugging purpose to linux-next, but I think any of these changes should be temporary in nature and have acknowledgement from maintainers of touched code. Perhaps these changes should also be in a separate topic branch which gets pulled from linux-next temporarily.
Quoted 7 lines
Quoted 3 lines
I'm happy if we can assign a git tree and a ML that are intended for temporary patches. But please note that asking fuzzers to test a tree for hosting temporary patches is not a choice. We need to test on massive instances to reproduce problems. linux.git is the
I don't understand why it's not an option, couldn't you just ask syzbot to apply the patches only for linux-next, or merge a tree but only for linux-next?
Basic constraint is
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-custom-patches .
I don't know whether
https://groups.google.com/g/syzkaller/c/4Y3zvo_t_lI/m/z-4-79ZHBQAJ
could loosen this constraint to some degree.
Quoted 4 lines
The point is not just syzbot consumes linux-next, everyone
does. Although I trust your good intents, but temporarily patches (especially when they are silently merged into linux-next) can undermine what linux-next is supposed to be.
I know that syzbot is not the only consumer of linux-next. Therefore, I try to minimize users affected (e.g. guard changes by CONFIG_DEBUG_AID_FOR_SYZBOT=y or CONFIG_KCOV=y ) when making changes.
Quoted 4 lines
By "silently" I mean I didn't see you gave a heads-up about doing this either, e.g. an email saying "hey I'm including these into the linux-next". Maybe I'm missing something here?
I'm fine with announcing to ML that I am about to make debug/temporary changes. But we might want a prefix for debug/temporary patches that are not intended for upstream? Also, we might want a kernel config option (e.g. CONFIG_DEBUG_AID_FOR_SYZBOT) for minimizing users affected?
Quoted 3 lines · Tetsuo Handa
Basic constraint is
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-custom-patches .
No, that section talks about two different things: patches that are fixes as well as temporary debugging patches.
For the former, it just says you should talk to maintainers to land the fix. That is: you need agreement from maintainers, which is what we have been saying.
For the latter, it simply says that sometimes debugging patches have been added in the past to linux-next. It doesn't say it was done unilaterally. (And even if it did, it is just a third-party document).
Critically, the section also explicitly ends with:
"One can also always run syzkaller locally on any kernel for better
stress testing
of a particular subsystem and/or patch."
...which sounds logical to me and is precisely what I was asking. So to me it seems you don't actually need to put it in linux-next.
Now, reading the Google Groups thread you link, it seems like your *actual constraint* doesn't come from syzkaller, but rather that you want to use the compute resources Google allocates for big trees like linux-next.
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
"Aleksandr says we can create a special mailing list to test draft patches, so the series sent to that list is fuzzed for some time, similarly to how upstream patch testing works now.
Do you think that would help you to debug these issues? "
So what happened with that? It seems there was a way forward there, no? (Cc'ing Alexander from that thread)
Cheers,
Miguel
Quoted 4 lines · Miguel Ojeda
Now, reading the Google Groups thread you link, it seems like your *actual constraint* doesn't come from syzkaller, but rather that you want to use the compute resources Google allocates for big trees like linux-next.
Note that if you are trying to debug a specific syzbot bug, you don't need to send it to linux-next. You can just reply to reproducer with
#syz test: git://repo/address.git branch-or-commit-hash
So I think we need to distinguish between two cases. One is where you are trying to address specific syzkaller reproducer. The other is where you want to do a full set of fuzzing on a draft series. I'd argue that though, that this is something you want to do when the patch is almost ready to land upstream --- and at that point, putting in linux-next makes perfect sense.
I'm not really see the case where it's not quite ready to land upstream (and so you want the full set of testing, not just with syzbot, with other people testing linux-next), but you still want to get the full set syzkaller testing.
Do people think this is really such a common scenario?
- Ted
Quoted 10 lines · Theodore Tso
Quoted 4 lines · Miguel Ojeda
Now, reading the Google Groups thread you link, it seems like your *actual constraint* doesn't come from syzkaller, but rather that you want to use the compute resources Google allocates for big trees like linux-next.
Note that if you are trying to debug a specific syzbot bug, you don't need to send it to linux-next. You can just reply to reproducer with
#syz test:
git://repo/address.gitbranch-or-commit-hash
For bugs which have reliable reproducers, "#syz test:" will work. My debug printk() patches are for bugs which don't have reproducers (or reproducer is too unreliable to reproduce). I need large compute resources for debugging such bugs.
Quoted 11 lines · Miguel Ojeda
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
"Aleksandr says we can create a special mailing list to test draft patches, so the series sent to that list is fuzzed for some time, similarly to how upstream patch testing works now.
Do you think that would help you to debug these issues? "So what happened with that? It seems there was a way forward there, no? (Cc'ing Alexander from that thread)
My response on that proposal is
https://groups.google.com/g/syzkaller/c/4Y3zvo_t_lI/m/jqRBCN5bBQAJ . In short, if trees with custom patches were tested with as much resources and many patterns/testcases as trees without custom patches, I won't need to carry custom patches in linux-next. But I think that we can't afford allocating so much resources for trees with custom patches.
I have carried custom patches in linux-next and it resulted in 10 patches for https://syzkaller.appspot.com/bug?extid=881d65229ca4f9ae8c84 where "#syz test:" did not work. Since remaining bugs are no longer easily reproducible in linux-next, I want to carry custom patches in the networking trees. But in general, git trees are not supposed to make "git reset --hard" changes.
Therefore, I am using my git tree as if "quilt" for making "git reset --hard" changes in linux-next tree. If a kernel config option which is used for fuzz testing were available in upstream, it will make easier to manage custom patches for temporary debugging and pinpoint-blocking stupid operations (like SysRq-b).
Quoted 3 lines · Tetsuo Handa
My response on that proposal is
https://groups.google.com/g/syzkaller/c/4Y3zvo_t_lI/m/jqRBCN5bBQAJ .
I am aware -- I read the thread before I replied. I was asking what happened after the thread, i.e. what was the outcome, if some more discussion took place elsewhere, etc.
Quoted 4 lines
In short, if trees with custom patches were tested with as much resources and many patterns/testcases as trees without custom patches, I won't need to carry custom patches in linux-next. But I think that we can't afford allocating so much resources for trees with custom patches.
Why? Just to avoid confusing myself: are you part of the team that decides that allocation? Or are you saying that they ended up saying it isn't possible to do so?
If these patches are so important, then the question is why those resources cannot be committed, or why cannot linux-next testing be reduced temporarily when there are such special runs going on.
Quoted 6 lines
I have carried custom patches in linux-next and it resulted in 10 patches for https://syzkaller.appspot.com/bug?extid=881d65229ca4f9ae8c84 where "#syz test:" did not work. Since remaining bugs are no longer easily reproducible in linux-next, I want to carry custom patches in the networking trees. But in general, git trees are not supposed to make "git reset --hard" changes.
I don't follow. If it were to be agreed that linux-next is the right place for those tests, then those patches could simply be extra branches that Mark could merge at the end -- no need to git reset the "normal" subsystem branches.
In other words, just like you are doing with your branch, essentially. But it is something that needs to be agreed upon first.
Quoted 4 lines
Therefore, I am using my git tree as if "quilt" for making "git reset --hard" changes in linux-next tree. If a kernel config option which is used for fuzz testing were available in upstream, it will make easier to manage custom patches for temporary debugging and pinpoint-blocking stupid operations (like SysRq-b).
The kernel config option is orthogonal, in my view. That is, even if the option were to be added, you would still need agreement to add random patches for testing (including discussing when in the cycle is best to do so, e.g. last time you added patches just before -rc7 which is about the worst time to do so, and which did break rust-next).
Cheers,
Miguel
Quoted 2 lines · Miguel Ojeda
is about the worst time to do so, and which did break rust-next).
s/rust-next/the rustdoc and Clippy clean builds in linux-next/
Cheers,
Miguel
Mark Brown
maintainerQuoted 7 lines · Miguel Ojeda
I don't follow. If it were to be agreed that linux-next is the right place for those tests, then those patches could simply be extra branches that Mark could merge at the end -- no need to
git resetthe "normal" subsystem branches.In other words, just like you are doing with your branch, essentially. But it is something that needs to be agreed upon first.
Yes, this is a good approach. Branches can be added or removed pretty easily so there's no problem on my end - I add temporary branches relatively often, there's currently one for Uwe's device ID reworks for example. Sometimes trees are managed in a way that makes it easy for the tree to do this locally too (eg, arm-soc merges all their topic branches themselves) so the maintainer might be happy to add things to their tree but if not then a temporary branch is a good approach.
If for some reason there's some difficulty in working with the relevant maintainers (eg, they're just not responding to mail at all) the process of asking for a branch to be added to -next is also a good way of making people aware that there's some issues and having some degree of coordination.
Mark Brown
maintainerQuoted 5 lines · Miguel Ojeda
On Sat, Jul 4, 2026 at 12:15 PM Tetsuo Handa
For the latter, it simply says that sometimes debugging patches have been added in the past to linux-next. It doesn't say it was done unilaterally. (And even if it did, it is just a third-party document).
...
Quoted 3 lines
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
Right, this seems to be something where people need to talk more. It does seem like there's real communication problems that need to be addressed, it looks like some progress is being made thanks to this thread.
Just dropping patches into -next that people are unaware of is going to cause issues sooner or later. Either there'll be some direct collision with other work or (probably worse) they'll cause someone to think that something is working better than it actually is since people testing -next are testing with some extra stuff that the maintainers weren't aware of. It might be that the most sensible thing is to have some extra patches in there for a period, it's an expected part of the flow that things might get backed out after all, but we want to avoid surprises.
Quoted 4 lines
Quoted 4 lines
I'm fine with announcing to ML that I am about to make debug/temporary changes. But we might want a prefix for debug/temporary patches that are not intended for upstream? Also, we might want a kernel config option (e.g.
CONFIG_DEBUG_AID_FOR_SYZBOT) for minimizing users affected?
TBH I think a prefix is probably not needed since there should be a bit more communciation about what's going on than just posting the patch - some discussion about how to debug whatever's going on for example.
Quoted 9 lines · Mark Brown
Quoted 5 lines · Miguel Ojeda
On Sat, Jul 4, 2026 at 12:15窶ッPM Tetsuo Handa
For the latter, it simply says that sometimes debugging patches have been added in the past to linux-next. It doesn't say it was done unilaterally. (And even if it did, it is just a third-party document).
...
Quoted 8 lines
Quoted 3 lines
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
Right, this seems to be something where people need to talk more. It does seem like there's real communication problems that need to be addressed, it looks like some progress is being made thanks to this thread.
Yes, thanks to this thread. There is real communication problems. Developers / maintainers are too busy to continue threads on my patches. Let's see several examples.
Example 1: (not a problem with linux-next only patch, but a communication problem)
Linus's "None of this makes any sense." comment at
https://lkml.kernel.org/r/CAHk-=wjmFiptPgaPx9vY3RG=rqO452UmOAPb1y_f9GQBtuJVjg@mail.gmail.com
made Mikulas to say "I don't know what else to do." at
https://lkml.kernel.org/r/[email protected] , causing
https://lkml.kernel.org/r/[email protected] and
https://lkml.kernel.org/r/[email protected] to stall. We need clarification from Linus or response from Mikulas.
Example 2: (a problem with moving from linux-next only patch to networking patch, but a communication problem)
I responded to Jakub's "Have you see netdev_hold() / netdev_put() ?" comment at
https://lkml.kernel.org/r/[email protected] but got no response. I need clarification or response from Jakub. If some core maintainer asks network people "Let's update all existing dev_put()/dev_hold() users to use netdev_put()/ netdev_hold()", some progress would be made. No progress as long as no response.
Example 3: (a problem with linux-next only patch, but a communication problem)
I responded to Al's "Not a peep in the logs, breakage still there" comment at
https://lkml.kernel.org/r/[email protected]
but got no response. Therefore, I sent an updated version
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=3ae78a69d3a9a4ad1d3b267324bd742654f40965
to linux-next, and I am currently waiting for syzbot to reproduce this problem.
Mark Brown
maintainerQuoted 5 lines · Tetsuo Handa
Quoted 1 line · Mark Brown
On Sat, Jul 04, 2026 at 02:06:43PM +0200, Miguel Ojeda wrote:
Example 1: (not a problem with linux-next only patch, but a communication problem)
Quoted 7 lines
Linus's "None of this makes any sense." comment at
https://lkml.kernel.org/r/CAHk-=wjmFiptPgaPx9vY3RG=rqO452UmOAPb1y_f9GQBtuJVjg@mail.gmail.com
made Mikulas to say "I don't know what else to do." at
https://lkml.kernel.org/r/[email protected] , causing
https://lkml.kernel.org/r/[email protected] and
https://lkml.kernel.org/r/[email protected] to stall. We need clarification from Linus or response from Mikulas.
That's less than a week old, and isn't it part of a bigger discussion that was going on between the filesystem people and the fuzzer people about how much to trust the data on disks?
Quoted 7 lines
Example 2: (a problem with moving from linux-next only patch to networking patch, but a communication problem)
I responded to Jakub's "Have you see
netdev_hold()/netdev_put()?" comment at
https://lkml.kernel.org/r/[email protected] but got no response. I need clarification or response from Jakub. If some core maintainer asks network people "Let's update all existing dev_put()/dev_hold() users to use netdev_put()/netdev_hold()", some progress would be made. No progress as long as no response.
That one seems like you need to follow up, it's been a quite a while. Probably mentioning the issues with netdev_hold()/put() (you said there were some, but I can't tell from the thread what they are) and the patches you want in one mail so things are more directly actionable.
Quoted 7 lines
Example 3: (a problem with linux-next only patch, but a communication problem)
I responded to Al's "Not a peep in the logs, breakage still there" comment at
https://lkml.kernel.org/r/[email protected]
but got no response. Therefore, I sent an updated version
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=3ae78a69d3a9a4ad1d3b267324bd742654f40965
to linux-next, and I am currently waiting for syzbot to reproduce this problem.
Again, that one seems fairly new (a bit staler than the first one). We did have the merge window though, some people drop everything when that passes.
It does look like you're seeing some stuff getting dropped on the floor here but TBH it doesn't look super unusual. There is a bit of an issue in our processes with things getting stuck in a quite submitter unfriendly way so you do often need to keep pushing on things.
Quoted 11 lines · Miguel Ojeda
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
"Aleksandr says we can create a special mailing list to test draft patches, so the series sent to that list is fuzzed for some time, similarly to how upstream patch testing works now.
Do you think that would help you to debug these issues? "So what happened with that? It seems there was a way forward there, no? (Cc'ing Alexander from that thread)
Alexander, was there any progress?
Quoted 11 lines · Tetsuo Handa
Quoted 9 lines · Miguel Ojeda
I understand it may be nice to use such resources, but the solution is to talk to both sides, as you do in that thread. From what I see, they even offered to add a mailing list for those patches:
"Aleksandr says we can create a special mailing list to test draft patches, so the series sent to that list is fuzzed for some time, similarly to how upstream patch testing works now.
Do you think that would help you to debug these issues? "
Quoted 5 lines
Quoted 2 lines
So what happened with that? It seems there was a way forward there, no? (Cc'ing Alexander from that thread)
Alexander, was there any progress?
Hi Tetsuo,
You can send patches to [email protected].
E.g. I ran
$ git send-email --suppress-cc=all [email protected] patches/kcov-tetsuo-v7/0001-kcov-fix-data-corruption-and-race-conditions-on-PREE.patch
to test your latest kcov patch and got the results here:
https://ci.syzbot.org/series/33930a89-bb2a-4c5a-b6c5-d45829d5c94c
The fuzzing ran on 8 VMs for 3 hours, and there were no new reports.
Please let us know what you think of this workflow.
Quoted 9 lines · Alexander Potapenko
Hi Tetsuo,
You can send patches to [email protected].
E.g. I ran
$ git send-email --suppress-cc=all [email protected] patches/kcov-tetsuo-v7/0001-kcov-fix-data-corruption-and-race-conditions-on-PREE.patch
Quoted 6 lines
to test your latest kcov patch and got the results here:
https://ci.syzbot.org/series/33930a89-bb2a-4c5a-b6c5-d45829d5c94cThe fuzzing ran on 8 VMs for 3 hours, and there were no new reports.
Please let us know what you think of this workflow.
Unfortunately, 8 VMs for 3 hours would be too little to stop my linux-next only patches.
syzbot took more than one month in linux-next until I was able to report
https://lkml.kernel.org/r/[email protected] . I could have reported within 24 hours if all VMs (I don't know how many VMs are assigned) for networking trees were tested with my linux-next only patch.
syzbot was not able to trigger
"BUG: %s/%u is doing I/O request on loop%d in Lo_rundown state.\n" message using https://lkml.kernel.org/r/[email protected]
in linux-next tree. syzbot could have triggered this message within 24 hours if all VMs for upstream trees were tested with my linux-next only patch.
What I want is a workflow that can test with patches for many days and many VMs.
Quoted 16 lines · Tetsuo Handa
Quoted 14 lines · Alexander Potapenko
Hi Tetsuo,
You can send patches to [email protected].
E.g. I ran
$ git send-email --suppress-cc=all [email protected] patches/kcov-tetsuo-v7/0001-kcov-fix-data-corruption-and-race-conditions-on-PREE.patchto test your latest kcov patch and got the results here:
https://ci.syzbot.org/series/33930a89-bb2a-4c5a-b6c5-d45829d5c94cThe fuzzing ran on 8 VMs for 3 hours, and there were no new reports.
Quoted 8 lines
Quoted 1 line
Please let us know what you think of this workflow.
Unfortunately, 8 VMs for 3 hours would be too little to stop my linux-next only patches.
syzbot took more than one month in linux-next until I was able to report
https://lkml.kernel.org/r/[email protected] . I could have reported within 24 hours if all VMs (I don't know how many VMs are assigned) for networking trees were tested with my linux-next only patch.
Currently, two syzbot instances point at linux-next:
ci-upstream-linux-next-kasan-gce-root (10 VMs) and ci-upstream-rust-kasan-gce (4 VMs).
So, we would need 420 VM-days of fuzzing to find that bug. Adding the 20 machines currently fuzzing net and net-next (assuming those are the ones you refer to) would have reduced the time to 12.5 days, but not to 24 hours.
During that time, we would stop fuzzing the main net/net-next repositories. Another problem is that switching instances between different Git trees is not zero-cost; it requires manual reconfiguration and introduces maintenance overhead.
Now imagine that requests to extensively fuzz with a certain debug patch arrive every week.
How do we decide which of those deserve more attention from the fuzzer? For how long should we keep those patches?
How do we ensure these patches do not conflict with each other?
Quoted 8 lines
syzbot was not able to trigger
"BUG: %s/%u is doing I/O request on loop%d inLo_rundownstate.\n" message using https://lkml.kernel.org/r/[email protected]
in linux-next tree. syzbot could have triggered this message within 24 hours if all VMs for upstream trees were tested with my linux-next only patch.What I want is a workflow that can test with patches for many days and many VMs.
I am afraid the only way to achieve that is to ensure the patch reaches upstream.
I could imagine a compromise: a community-maintained repository containing a bunch of debug patches applied on top of linux-next. But we won't be able to allocate too many resources to fuzzing that repository (it would be just one of the many), so reproducing certain bugs may still take weeks.
Quoted 2 lines · Alexander Potapenko
During that time, we would stop fuzzing the main
net/net-nextrepositories.
I don't want to stop the main repositories.
Quoted 2 lines
I am afraid the only way to achieve that is to ensure the patch reaches upstream.
Yes, I know. I think there are two problems (technical one and procedural one).
The technical problem is that adding debug patches can impact performance (especially when printk() is called frequently). We have CONFIG_KCOV in the upstream kernels but we don't have something like CONFIG_DEBUG_AID_FOR_SYZBOT in the upstream kernels. If we can assume that kernels built with CONFIG_KCOV=y are not subjected to performance regression tests, I can try to minimize impact by guarding with CONFIG_KCOV=y. But if we can't assume, I want a switch like CONFIG_DEBUG_AID_FOR_SYZBOT.
The procedural problem is that developers / maintainers are too busy to respond to my debug patches. Since I am debugging difficult bugs where nobody else are willing to spend resources, this is an expected result. I am OK to post my debug patches to ML. But without responses / interests from developers / maintainers, this problem won't be able to make progress.
Just saying "Not my business." is not helpful.
Quoted 8 lines · Tetsuo Handa
The procedural problem is that developers / maintainers are too busy to respond to my debug patches. Since I am debugging difficult bugs where nobody else are willing to spend resources, this is an expected result. I am OK to post my debug patches to ML. But without responses / interests from developers / maintainers, this problem won't be able to make progress.
Just saying "Not my business." is not helpful.
Is what is happening that when your security LSM (Tomoyo) is enabled, this causes lockdep or warnings or other issues in other subsystems?
If so the question is how much are other developers and maintainers obliged to help you. The historical answer has been that we help each other out as a matter of professional courtesy, and if someone has helped me out a lot, I will tend to return the favor when they ask me for my help. I've even used that as an argument for why a company should dedicate some amount of their developer's time helping out upstream code even if it's not directly related to their company's direct interest, since that contribution builds up karma that may cause the upstream maintainers to help debug a bug, or upstream some feature, that the company *really* cares about.
But there is nothing which obliges other developers / maintainers to help you out, other than their being nice. This is another way of saying that "free software" is free as in freedom, and not free as in "free beer". Responding to your debug patches is in effect demanding free support. If you can convince them that it is in their interest, or their company's interest to help you debug your issue, then that's fine. But maybe there's a good *reason* why no one is willing to spend resources debugging a particular issue --- namely, that it doesn't help them or their company?
Regards,
- Ted
Quoted 8 lines · Theodore Tso
Quoted 7 lines · Tetsuo Handa
The procedural problem is that developers / maintainers are too busy to respond to my debug patches. Since I am debugging difficult bugs where nobody else are willing to spend resources, this is an expected result. I am OK to post my debug patches to ML. But without responses / interests from developers / maintainers, this problem won't be able to make progress.
Quoted 4 lines
Quoted 1 line
Just saying "Not my business." is not helpful.
Is what is happening that when your security LSM (Tomoyo) is enabled, this causes lockdep or warnings or other issues in other subsystems?
No. Problems are not related to tomoyo.
What is happening is that I am using tomoyo git tree as a hook for executing debug printk() (using linux-next git tree) in order to debug problems which syzbot has found but nobody can afford looking into.
Individual git tree is not served as a hook for applying debug patches like "quilt push"; one of reasons is that "git" (compared to "quilt") is not intended for removing a patch when that patch became no longer necessary (due to problem being fixed).
Quoted 9 lines · Tetsuo Handa
What is happening is that I am using tomoyo git tree as a hook for executing debug
printk()(using linux-next git tree) in order to debug problems which syzbot has found but nobody can afford looking into.Individual git tree is not served as a hook for applying debug patches like "quilt push"; one of reasons is that "git" (compared to "quilt") is not intended for removing a patch when that patch became no longer necessary (due to problem being fixed).
Ah. In that case, what if you simply have a separate tree which just has these debugging patches, and then either Mark Brown or the Syzbot team could create a new git tree / branch which takes linux-next and then merges this debug tree on top of the latest linux-net?
The advantages of this is that the patches never hit linux-next, but this new tree, call it linux-next-syz-debug. And since it is a new tree, you wouldn't need a new magic KConfig CONFIG #ifdef, so no code changes would be needed in syzbot.
The only question then is how much does the Syzbot team is willing pay to help provide the resources you need to resolve those syzbot issues. I will say that *my* particular management chain at $WORK has authorized ignoring a whole class of syzbot bugs because they are utterly irrelevant for our production environment, and so it's not so much a "can't afford", but "has insufficient business value relative to our budget / head count authorization". (And personally, I refuse to work on those bugs after midnight or on weekends; I have higher priority personal priorities.) But perhaps the Syzbot team has a different set of priorities....
- Ted
Quoted 11 lines · Theodore Tso
Quoted 9 lines · Tetsuo Handa
What is happening is that I am using tomoyo git tree as a hook for executing debug
printk()(using linux-next git tree) in order to debug problems which syzbot has found but nobody can afford looking into.Individual git tree is not served as a hook for applying debug patches like "quilt push"; one of reasons is that "git" (compared to "quilt") is not intended for removing a patch when that patch became no longer necessary (due to problem being fixed).
Quoted 4 lines
Ah. In that case, what if you simply have a separate tree which just has these debugging patches, and then either Mark Brown or the Syzbot team could create a new git tree / branch which takes linux-next and then merges this debug tree on top of the latest linux-net?
Yes, how to apply debug printk() patch will be solved if there were git trees which applies debug patches on top of linux.git , linux-next.git , net.git etc.
But there is a potential problem. Developers might not try to look into downstream git trees. How many developers would "git clone" when syzbot reports an issue found in git trees with debug patches applied?
Quoted 2 lines
The only question then is how much does the Syzbot team is willing pay to help provide the resources you need to resolve those syzbot issues.
I wonder whether it is appropriate to call "syzbot issues". syzbot is finding bugs in upstream git trees. Unless the cause of a bug is either syzbot is doing a stupid thing or compiler/toolchain is broken, it is "an upstream kernel issue".
Quoted 5 lines
I will say that *my* particular management chain at $WORK has authorized ignoring a whole class of syzbot bugs because they are utterly irrelevant for our production environment, and so it's not so much a "can't afford", but "has insufficient business value relative to our budget / head count authorization".
Yes, that is the biggest issue.
Those who hate my attempts (i.e. sending patches to linux-next without being reviewed/acked by maintainers) are expecting that any patch is reviewed/acked by maintainers. But some maintainers are considering bugs reported by syzbot as "not worth looking into". Therefore, not only debug printk() patches (for bugs found by syzbot) but also bug fix patches (which syzbot has found) are getting ignored.
There are many patches which are not promptly applied into relevant git trees. For example, syzbot is still using linux-next-20260714
( https://syzkaller.appspot.com/upstream/manager/ci-upstream-linux-next-kasan-gce-root ) despite the latest linux-next tree is linux-next-20260723 because e.g.
https://lore.kernel.org/all/al1pElMQZsDfpAYI@michalis-linux/T/ is not yet applied. There is little effort in the community that "we try to make it possible to continue fuzzing the linux-next.git tree in order to find new bugs as early as possible and fix new bugs before it reaches linux.git tree".
In other words, the interest (by the upstream Linux kernel community) on fixing kernel bugs which syzbot found are decreasing. This is a deadlock situation, and just saying "Not my business." is not helpful.
Quoted 4 lines · Tetsuo Handa
I wonder whether it is appropriate to call "syzbot issues". syzbot is finding bugs in upstream git trees. Unless the cause of a bug is either syzbot is doing a stupid thing or compiler/toolchain is broken, it is "an upstream kernel issue".
I know that people have different experiences of Syzbot reported issues, but the bugs I see require root privileges, modifying a file system while it is mounted, maliciously corrupting file system images and then mounting it without first running fsck, mucking with the loop device (which requires root privileges) while it is in use by a file system.
I do agree that if you mount a file system without running fsck, and it has been maliciously corrupted, it woud be nice(tm) if it didn't result in a crash or a WARN_ON. If someone wants to fund the engineering work to make this be true, so long as it doesn't compromise long-term code maintainability or performance, patches would be welcome. So perhaps this is a P2 bug. But it's not something I'm going to consider high priority, because it's not in any threat model that I or the people who manage the budget and head count authorizations, care about.
Since that's the *vast* majority of the issues reported by syzbot, I tend to use "syzbot bug" as a shorthand.
I can't speak to whether the syzbot bugs you are chasing fall into this category or not. But if it requires *vast* amount of VM time, it makes me suspect that it falls into that category.
Quoted 6 lines
Those who hate my attempts (i.e. sending patches to linux-next without being reviewed/acked by maintainers) are expecting that any patch is reviewed/acked by maintainers. But some maintainers are considering bugs reported by syzbot as "not worth looking into". Therefore, not only debug
printk()patches (for bugs found by syzbot) but also bug fix patches (which syzbot has found) are getting ignored.
Yep, and I've explained above why that's the case. I've tried to make the case that if there was a way that we could easily separate out those bugs which are signal from noise (at leat as defined by those people who refuse to fund this development work), it might be good for improving syzbot's reputation. But I'm so busy that I don't have time to go through all of the syzbot reported issues to see if any of them are actually ones that should be considered high priority.
Quoted 3 lines
In other words, the interest (by the upstream Linux kernel community) on fixing kernel bugs which syzbot found are decreasing. This is a deadlock situation, and just saying "Not my business." is not helpful.
Perhaps if Syzbot could be improved so we can be more selective about which bugs are worth fixing, it might actually be better for the kernel's quality.
But if the goal is to get ourselves to zero bugs, or zero syzbot reported issues, that's not a goal I can get behind. It certainly isn't something my management chain isn't willing to support that goal, and it's NOT going to happen out of my own personal time.
Cheers,
- Ted
Quoted 7 lines · Theodore Tso
Quoted 3 lines
In other words, the interest (by the upstream Linux kernel community) on fixing kernel bugs which syzbot found are decreasing. This is a deadlock situation, and just saying "Not my business." is not helpful.
Perhaps if Syzbot could be improved so we can be more selective about which bugs are worth fixing, it might actually be better for the kernel's quality.
One non-critical bug in some module can prevent other critical bugs in different modules from being discovered. We can't expect that syzbot can find only critical bugs, without an effort to fix non-critical bugs.
Quoted 5 lines
But if the goal is to get ourselves to zero bugs, or zero syzbot reported issues, that's not a goal I can get behind. It certainly isn't something my management chain isn't willing to support that goal, and it's NOT going to happen out of my own personal time.
https://www.youtube.com/watch?v=qrBVXxZDVQY or
https://events19.linuxfoundation.org/wp-content/uploads/2017/11/Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf
says that the goal is "Reduce bugs/release 100x: 20'000 -> 200".
I know that individual maintainer cannot afford looking into all patches that involve syzbot. A maintainer has the permission to accept/reject patches on an area he/she maintains. The problem is that a maintainer who has the permission to accept/reject patches tends to become defunctional when syzbot is involved. What actions can the Linux kernel community take in order to address this problem (instead of just ignoring developers who are trying to fix bugs)?
Quoted 5 lines · Tetsuo Handa
The problem is that a maintainer who has the permission to accept/reject patches tends to become defunctional when syzbot is involved. What actions can the Linux kernel community take in order to address this problem (instead of just ignoring developers who are trying to fix bugs)?
Fundamentally, not all bugs are worth fixing. And if you believe that all bugs are worth fixing, you need to convince companies to **fund** that work, and not leave it as unfunded mandate. If you give me some minions who can be dedicated to help fix bugs, I'm certainly willing to provide technical direction. But there's no such thing as a free lunch --- and the business case is not as clear cut as you seem to be assuming.
In essence, this is a cost allocation problem. When you ask for help from other maintainers to fix something that you care about, if it's not important to them or their company / management chain, you are dumping costs from you to them. This can work if you can also help them in turn in other areas. This happens all the time, actually, and it's what makes Open Source work.
So I think the way we solve this problem is by finding ways to allocate the costs to those companies or organizations that think it is a good fit for their business, and to reduce the burden on those whose company (or management chain) doesn't think the business case is sufficiently compelling.
Other solutions might include finding some AI solution which can filter the low-priority quality Syzbot reports from the high-quality ones. If some companies or some departments may have different priorities, perhaps they can use different filuters, so they can work on those bugs that they think are important.
Perhaps AI's can do a better job at proposing some patches that resolve the problem, or creating reliable reproducers that don't require thousands of VM hours of testing.
One of the reasons why I am personally more supportive about Sashiko than syzbot is because Sashiko is a technology which is reducing *my* workload. It does transfer more work to the contributor, but if they are trying to get code into the kernel and if their company think there is a business case for that new feature, it seems right that we require that pay the development costs of submitting high-quality code.
Cheeres,
- Ted
Quoted 7 lines · Theodore Tso
Quoted 5 lines · Tetsuo Handa
The problem is that a maintainer who has the permission to accept/reject patches tends to become defunctional when syzbot is involved. What actions can the Linux kernel community take in order to address this problem (instead of just ignoring developers who are trying to fix bugs)?
Quoted 7 lines
Fundamentally, not all bugs are worth fixing. And if you believe that all bugs are worth fixing, you need to convince companies to **fund** that work, and not leave it as unfunded mandate. If you give me some minions who can be dedicated to help fix bugs, I'm certainly willing to provide technical direction. But there's no such thing as a free lunch --- and the business case is not as clear cut as you seem to be assuming.
Have you ever seen https://syzkaller.appspot.com/upstream/syz-dungeon which ranks contributors (kernel developers) based on their achievements in fixing bugs found by Syzkaller? You can find that I am the second top contributor.
Also, my activity for fixing bugs found by syzbot is not my $WORK. I am working on those bugs using my private time. I am neither working at Google-related companies nor a member of syzbot team (who can change kernel config for syzbot kernels in order to help syzbot continue testing).
Quoted 7 lines
In essence, this is a cost allocation problem. When you ask for help from other maintainers to fix something that you care about, if it's not important to them or their company / management chain, you are dumping costs from you to them. This can work if you can also help them in turn in other areas. This happens all the time, actually, and it's what makes Open Source work.
That is wrong. I make not only debug printk() patches but also bug-fix patches. I am asking for approval from maintainers to fix bugs that syzbot found. It is really annoying that maintainers keep silence for unknown reason (maybe they are thinking that "I don't care about this bug"). I can't make forward progress if maintainers keep silence. What are the maintainers for?
At the panel discussion in LinuxCon Europe 2011, Linus Torvalds has said "the average age at the summit has risen by one year every year" ( https://lwn.net/Articles/464530/ ). If maintainers are there to let developer's effort be sent to /dev/null, trying to contribute is waste of time and new developers won't grow up.
Quoted 6 lines
So I think the way we solve this problem is by finding ways to allocate the costs to those companies or organizations that think it is a good fit for their business, and to reduce the burden on those whose company (or management chain) doesn't think the business case is sufficiently compelling.
Not all developers have enough money to pay to companies/organizations. Some developers (including me) are contributing by spending their time instead of paying money.
Your statements sound like "Poor men/women cannot participate into the Linux kernel development." Please don't assume that everything is controlled by "who pays the money?".
Quoted 6 lines
Other solutions might include finding some AI solution which can filter the low-priority quality Syzbot reports from the high-quality ones. If some companies or some departments may have different priorities, perhaps they can use different filuters, so they can work on those bugs that they think are important.
syzbot already has a priority filter. But trying to make patches based on that filter will result in a waste of time if maintainers are not responding.
Please respond to syzbot-related patches. For example, if you are not interested in a bug syzbot has reported, respond to that report like
"I do not think that this bug is important enough to fix. Therefore,
I won't give you Reveiwed-by: nor Acked-by: even if you propose a patch."
. Then, the developers can consider what to do. For example, use a different ML which handles unloved patches, and make it possible to send the proposed patches to linux-next and linux without Reveiwed-by: or Acked-by: from maintainers, unless Sashiko finds a new problem introduced by the proposed patches.
Current situation (no response from developers / maintainers) is a deadlock.
Quoted 7 lines · Tetsuo Handa
Not all developers have enough money to pay to companies/organizations. Some developers (including me) are contributing by spending their time instead of paying money.
Your statements sound like "Poor men/women cannot participate into the Linux kernel development." Please don't assume that everything is controlled by "who pays the money?".
If they want to spend their own personal time, that's fine. I just don't want companies or hobbists spending *my* personal time.
Quoted 1 line
Please respond to syzbot-related patches.
Oh, I, or some other ext4 developer do respond --- if it's a bug fix. I wish more of the contributors would actually pay attention to the Sashiko reviews, and actually *test* the patches using "kvm-xfstests smoke" which takes 20 minutes, but we'll respond, one way or another.
We might NACK the patch if it destroys performance because it adds tests in a hot path and it's addressing a really pointless syzbot scenario which requires root privilegs to modify the loop device while the file system is mounted, however.
However, if it's a debugging patch to fix what we might consider a rather pointless syzbot bug (and there are an awful lot of those, at least for ext4) it might take a while for me to respond. And if it's about landing ext4 patches from a non-ext4 tree, abusing the linux-next tree, that's a somewhat different matter....
- Ted
Quoted 4 lines · Tetsuo Handa
Those who hate my attempts (i.e. sending patches to linux-next without being reviewed/acked by maintainers) are expecting that any patch is reviewed/acked by maintainers. But some maintainers are considering bugs
It is not "hate", it is just the linux-next rules:
"
* submitted under GPL v2 (or later) and include the Contributor's Signed-off-by,
* posted to the relevant mailing list,
* reviewed by you (or another maintainer of your subsystem tree),
* successfully unit tested, and
* destined for the current or next Linux merge window.
"
That is why nobody is expecting random patches to appear there -- because every maintainer receives those rules when they add their tree to linux-next.
And those rules make sense -- adding patches that are not ready or unexpected can confuse others, break the build, add conflicts or, worse of all, silently invalidate testing of normal patches going to Linus.
Now, again, exceptions are definitely possible (e.g. I asked for one to add Rust many months before it was finally merged), but one needs to be quite careful and for something treewide like what you are asking, it will always be tricky...
E.g. even if you always managed to perfectly place the changes behind a CONFIG_, subsystems should probably still be warned, and Mark would probably need to drop your side if there are conflicts to be safe unless you coordinated the resolution beforehand with the maintainers etc.
Cheers,
Miguel
Quoted 18 lines · Miguel Ojeda
Quoted 4 lines · Tetsuo Handa
Those who hate my attempts (i.e. sending patches to linux-next without being reviewed/acked by maintainers) are expecting that any patch is reviewed/acked by maintainers. But some maintainers are considering bugs
It is not "hate", it is just the linux-next rules:
"
* submitted under GPL v2 (or later) and include the Contributor's Signed-off-by,
* posted to the relevant mailing list,
* reviewed by you (or another maintainer of your subsystem tree),
* successfully unit tested, and
* destined for the current or next Linux merge window.
"
Quoted 3 lines
That is why nobody is expecting random patches to appear there -- because every maintainer receives those rules when they add their tree to linux-next.
Excuse me, but what does "successfully unit tested" mean? Broken patches that do not build, or trigger oops by just "/bin/cat" _are_ arriving at linux-next tree, which in turn preventing continuous testing by syzbot.
e.g. https://lkml.kernel.org/r/[email protected]
(and I was using
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?h=next-20260702&id=110ef9f94f84d425ab55d09b2d70ed12ceffdf08
in order to help syzbot continue testing)
e.g. https://lkml.kernel.org/r/[email protected]
(and I was using a linux-next only patch which disables this module for syzbot kernels
(link to this patch already gone because this patch was more than 3 months ago) in order to help syzbot continue testing)
Also, WARN*() is abused just like printk() for catching oversights rather than introducing grace period for confirmation, which is also preventing continuous testing by syzbot. Sadly, developers/maintainers do not think that WARN*() is effectively panic() for continuous testing systems. Unless developers/maintainers can apply all necessary fixes within few days, WARN*() should not be used.
e.g. https://lkml.kernel.org/r/[email protected]
(and I was not able to use a linux-next only patch because WQ is a core component which cannot be disabled for syzbot kernels)
The latest one seems to be https://lore.kernel.org/all/al1pElMQZsDfpAYI@michalis-linux/T/
(but I am not able to use a linux-next only patch which disables this module
for syzbot kernels in order to help syzbot continue testing, for I currently have no route to help syzbot continue testing)
Quoted 5 lines
And those rules make sense -- adding patches that are not ready or unexpected can confuse others, break the build, add conflicts or, worse of all, silently invalidate testing of normal patches going to Linus.
I am OK to post my debug patches to the relevant mailing list. But without responses from developers / maintainers, my debug patches won't be able to find a route to linux-next, which in turn prevents us from debugging.
Quoted 4 lines · Tetsuo Handa
Excuse me, but what does "successfully unit tested" mean? Broken patches that do not build, or trigger oops by just "
/bin/cat" _are_ arriving at linux-next tree, which in turn preventing continuous testing by syzbot.
Yes, there are mistakes in linux-next all the time, and those mistakes take time away from maintainers. Which is why adding more unexpected patches makes it worse.
But, really, that is not the question at hand -- you say:
"my debug patches won't be able to find a route to linux-next, which
in turn prevents us from debugging."
No, what prevents you from debugging is that, from what you say, nobody is giving enough compute resources for your task.
linux-next is just a way to access some resources. Which may be OK or not to do so, but clearly it was not in agreement by everyone nor in alignment with what linux-next is supposed to be so far, and that is what I pointed out, nothing else.
Cheers,
Miguel
Quoted 2 lines · Miguel Ojeda
No, what prevents you from debugging is that, from what you say, nobody is giving enough compute resources for your task.
OK. Let's go back to https://lkml.kernel.org/r/[email protected] .
If I were able to use a kernel config option which minimizes impact for others, the technical problem for my task will be solved.
But including what kernel config options (or kernel command line parameters) I can use, there hardly is a discussion from developers / maintainers. The last discussion was 6 years ago:
https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]
Can we resume this discussion so that we can minimize surprising other developers / maintainers?
Boqun Feng
maintainerQuoted 4 lines · Tetsuo Handa
Quoted 2 lines · Miguel Ojeda
No, what prevents you from debugging is that, from what you say, nobody is giving enough compute resources for your task.
Quoted 8 lines
OK. Let's go back to https://lkml.kernel.org/r/[email protected] .
If I were able to use a kernel config option which minimizes impact for others, the technical problem for my task will be solved.
But including what kernel config options (or kernel command line parameters) I can use, there hardly is a discussion from developers / maintainers. The last discussion was 6 years ago:
As I said [1], the reposting was stopped 6 years ago, and no one could read other's mind to know whether it's still worth discussing.
Quoted 5 lines
https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]Can we resume this discussion so that we can minimize surprising other developers / maintainers?
Yes, resending the patches, talking about it in a conference, or giving people a heads-up when planning to queue it via linux-next is the way to resume the discussion. Could you please do any of these?
[1]: https://lore.kernel.org/lkml/akpxYEF2ZI7p1zn3@MacBook-0RXW5/
Regards,
Boqun
Mark Brown
maintainerQuoted 3 lines · Tetsuo Handa
Excuse me, but what does "successfully unit tested" mean? Broken patches that do not build, or trigger oops by just "
/bin/cat" _are_ arriving at linux-next tree, which in turn preventing continuous testing by syzbot.
You're supposed to make some reasonable effort to ensure that things work, if you break things badly enough and pay little enough attention your tree will get kicked. To even get merged in -next on a given day a tree has to build in some of the more common configurations, and not break KUnit. It's not expected that everything will be perfect, people can't be reasonably expected to to test all possible configurations - a big part of the point of -next is to pool coverage - but it's expected that people will be making a reasonable effort.
Quoted 3 lines
I am OK to post my debug patches to the relevant mailing list. But without responses from developers / maintainers, my debug patches won't be able to find a route to linux-next, which in turn prevents us from debugging.
The other side of this is that the reason people are noticing that you have added these patches is that you are causing breakage and disruption for them that catches them by surprise, a lot of what's coming through here is that you need to work actively with other people.
Quoted 4 lines · Mark Brown
The other side of this is that the reason people are noticing that you have added these patches is that you are causing breakage and disruption for them that catches them by surprise, a lot of what's coming through here is that you need to work actively with other people.
I will note that the last person who attempted to push changes to other subsystems via their own tree was Kent Overstreet. That was considered **extremely** anti-social, and after people complained Linus agreed to reject any pull request that had changes outside of bcachefs.
This is slightly different, in that it is pushing broken patches into linux-next, instead of Linus's tree, but I think there is a similar precedent. If someone is doing something abusive, it doesn't matter whether it's because they think they are doing something where the means justify the ends (whether it's "I fix syzbot bugs" or "I make a file system which doesn't eat your data"). If you do things in an antisocial way, Maintainers are *not* obligated to work with you, and while you can appeal to Linus, you might not like the resolution....
- Ted
Quoted 6 lines · Mark Brown
Quoted 3 lines · Tetsuo Handa
Excuse me, but what does "successfully unit tested" mean? Broken patches that do not build, or trigger oops by just "
/bin/cat" _are_ arriving at linux-next tree, which in turn preventing continuous testing by syzbot.
Quoted 8 lines
You're supposed to make some reasonable effort to ensure that things work, if you break things badly enough and pay little enough attention your tree will get kicked. To even get merged in -next on a given day a tree has to build in some of the more common configurations, and not break KUnit. It's not expected that everything will be perfect, people can't be reasonably expected to to test all possible configurations - a big part of the point of -next is to pool coverage - but it's expected that people will be making a reasonable effort.
You and I are talking about different topics.
You are saying that I should not submit patches that break linux-next. I am saying that maintainers are submitting patches that break linux-next (and then stop responding after a breakage was found in linux-next).
Some maintainers respond quickly (e.g. remove problematic commits from the tree) when syzbot sends a report that there was a build failure or boot failure with linux-next kernel, but other maintainers respond very slowly.
For the latter example, syzbot is still using next-20260714 because a maintainer who should accept/reject https://lkml.kernel.org/r/al1pElMQZsDfpAYI@michalis-linux
is not responding. As a result, syzbot is unable to test changes made between next-20260714 and next-20260726.
Since one of purposes for linux-next.git is "find bugs before bugs reach linux.git", being unable to follow daily next-YYYYMMDD is annoying. For example,
https://lkml.kernel.org/r/[email protected] (which caused a rust-next breakage while trying to find possible fix for this problem) was not discovered immediately because syzbot was unable to test next-202604{03,06,07,08,09,10} due to a different bug. This problem was introduced at the worst timing (i.e. unable to test around the merge window), and maintainers are not responding.
To minimize period of being unable to follow daily next-YYYYMMDD, I was sometimes sending linux-next only patches that disable specific module under CONFIG_DEBUG_AID_FOR_SYZBOT=y, for we can't predict when a maintainer responds to a build failure or boot failure in linux-next. But now that you want me to stop sending linux-next only patches that disable specific module under CONFIG_DEBUG_AID_FOR_SYZBOT=y, my question is what we can do to mitigate this "being unable to follow daily next-YYYYMMDD" problem.
Mark Brown
maintainerQuoted 11 lines · Tetsuo Handa
Quoted 9 lines · Mark Brown
Quoted 3 lines · Tetsuo Handa
Excuse me, but what does "successfully unit tested" mean? Broken patches that do not build, or trigger oops by just "
/bin/cat" _are_ arriving at linux-next tree, which in turn preventing continuous testing by syzbot.You're supposed to make some reasonable effort to ensure that things work, if you break things badly enough and pay little enough attention your tree will get kicked. To even get merged in -next on a given day a
Quoted 5 lines
You and I are talking about different topics.
You are saying that I should not submit patches that break linux-next. I am saying that maintainers are submitting patches that break linux-next (and then stop responding after a breakage was found in linux-next).
No, what I'm describing applies to everyone. It's a bit more important that if you're adding things not for your own tree that you don't mess up, but I do actually occasionally find myself removing trees because of disruption they are causing.
Quoted 8 lines
Some maintainers respond quickly (e.g. remove problematic commits from the tree) when syzbot sends a report that there was a build failure or boot failure with linux-next kernel, but other maintainers respond very slowly.
For the latter example, syzbot is still using next-20260714 because a maintainer who should accept/reject https://lkml.kernel.org/r/al1pElMQZsDfpAYI@michalis-linux
is not responding. As a result, syzbot is unable to test changes made between next-20260714 and next-20260726.
This really just looks like another case where you (or someone) need to commuicate if something is urgent - I would have a hard time telling from that thread that there's a particularly big problem, the only reply indicating any kind of problem with the posted patch (from Joshua) was then followed up saying to disrgard it and there's nothing at all at any point in the thread or original report about the scope of the issue or any practical impact it's having. Frankly I'm not clear that Michail (who posted the patch) is aware that there's any urgency here and isn't just acting reasonably and promptly on what was in the report mail.
Even if someone were to click through the syzbot link the report is flagged as low priority in the dashboard and the config is just a link to a raw config so there's no indication if this is defconfig, randconfig or whatever. There's absolutely nothing in the sysbot dashboard or list report that indicates to me that this has any kind of impact beyond "a warning is displayed during boot".
If this were something super urgent I would expect to see people following up saying clearly why that's the case, doing so repeatedly if things take a while. That simply isn't happening here, from what's visible I really don't know how anyone is expected to infer that this isn't just a fairly standard bugfix for a routine issue that was only reported a week ago. Perhaps this is all clear if you are familiar with how exactly syzbot works but most people aren't.
Quoted 5 lines · Mark Brown
Quoted 4 lines
For the latter example, syzbot is still using next-20260714 because a maintainer who should accept/reject https://lkml.kernel.org/r/al1pElMQZsDfpAYI@michalis-linux
is not responding. As a result, syzbot is unable to test changes made between next-20260714 and next-20260726.
Quoted 9 lines
This really just looks like another case where you (or someone) need to commuicate if something is urgent - I would have a hard time telling from that thread that there's a particularly big problem, the only reply indicating any kind of problem with the posted patch (from Joshua) was then followed up saying to disrgard it and there's nothing at all at any point in the thread or original report about the scope of the issue or any practical impact it's having. Frankly I'm not clear that Michail (who posted the patch) is aware that there's any urgency here and isn't just acting reasonably and promptly on what was in the report mail.
This problem is caused by commit 860e748bddcc ("drm: ensure blend mode supported
if pixel format with alpha exposed") at
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=860e748bddcc9291cbdd23e801640aeeba30cc44 .
That commit says "Drivers triggering this warning must be fixed.". That is, they know that it is expected that WARN*() fires. But they don't know that syzbot stops upon encountering "BUG:" or "WARNING:" within printk() output.
They also don't know that syzbot builds-in almost all modules into vmlinux. That is, initialization functions of most modules are executed upon boot. If one of modules emits "BUG:" or "WARNING:" line, the boot fails.
Maintainers sometimes apply commits that cause some in-tree modules to emit "BUG:" or "WARNING:" line, without having grace period for confirming that all in-tree modules are updated/fixed. Such decision is annoying for syzbot.
Quoted 7 lines
Even if someone were to click through the syzbot link the report is flagged as low priority in the dashboard and the config is just a link to a raw config so there's no indication if this is defconfig, randconfig or whatever. There's absolutely nothing in the sysbot dashboard or list report that indicates to me that this has any kind of impact beyond "a warning is displayed during boot".
That priority is determined based on how critical the bug is (e.g. use-after-free write). Therefore, being unable to build or boot is considered low priority because it can't cause e.g. use-after-free write. However, the real impact of "being unable to build or boot" is indicated at https://syzkaller.appspot.com/upstream/manager/ci-upstream-linux-next-kasan-gce-root . Currently syzbot is failing to update linux-next kernels for two weeks due to commit 860e748bddcc being in the linux-next tree.
All new bugs which are introduced while syzbot is unable to update linux-next kernels are reported only after syzbot succeeded to update linux-next kernels. The "being unable to build or boot" caused by addition of WARN*() can break whole testing by syzbot.
Mark Brown
maintainerQuoted 9 lines · Tetsuo Handa
Quoted 3 lines · Mark Brown
This really just looks like another case where you (or someone) need to commuicate if something is urgent - I would have a hard time telling from that thread that there's a particularly big problem, the only reply
That commit says "Drivers triggering this warning must be fixed.". That is, they know that it is expected that WARN*() fires. But they don't know that syzbot stops upon encountering "BUG:" or "WARNING:" within
printk()output.
Right, I think some of this is the difference between "needs to fix this" and "needs to fix this right now" - WARN_ON()s obviously need to be fixed, but it's not so obvious that it's a major disruption.
Quoted 3 lines
They also don't know that syzbot builds-in almost all modules into vmlinux. That is, initialization functions of most modules are executed upon boot. If one of modules emits "BUG:" or "WARNING:" line, the boot fails.
Oh, that *is* surprising - more people tend towards heavily modular kernels than fully built in ones, I suspect a lot of driver people won't realise that they're impacting anyone who doesn't have the hardware since it's not the norm.
Quoted 3 lines
All new bugs which are introduced while syzbot is unable to update linux-next kernels are reported only after syzbot succeeded to update linux-next kernels. The "being unable to build or boot" caused by addition of WARN*() can break whole testing by syzbot.
Indeed. I'd suggest following up on the fix with this information to try to chase things along.
Quoted 2 lines · Tetsuo Handa
syzbot stops upon encountering "BUG:" or "WARNING:" within
printk()output.
Out of curiosity, is that a fundamental property or something that could be optionally be relaxed (for certain linux-next runs only, perhaps) so that you can continue testing in certain cases where the "WARNING:" may be harmless (i.e. irrelevant for your particular fuzzing)?
i.e. I was wondering whether the system could ignore the "WARNING:"s (and avoid the kernel panicking on them etc.) until after boot, i.e. when the fuzzing starts.
Cheers,
Miguel
Quoted 11 lines · Miguel Ojeda
Quoted 2 lines · Tetsuo Handa
syzbot stops upon encountering "BUG:" or "WARNING:" within
printk()output.Out of curiosity, is that a fundamental property or something that could be optionally be relaxed (for certain linux-next runs only, perhaps) so that you can continue testing in certain cases where the "WARNING:" may be harmless (i.e. irrelevant for your particular fuzzing)?
Quoted 3 lines
i.e. I was wondering whether the system could ignore the "WARNING:"s (and avoid the kernel panicking on them etc.) until after boot, i.e. when the fuzzing starts.
syzbot uses kernel.panic_on_warn=1, for most of WARN*() users are reporting problems that should not be ignored. Therefore, WARN*() must not be used for reporting recoverable mistakes like -EINVAL. Abusing WARN*() like
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/kernel/workqueue.c?id=6e845bcb78c95af935094040bd4edc3c2b6dd784#n5855
is a fatal barrier for continuous testing systems like syzbot.
Quoting from https://lkml.kernel.org/r/[email protected] :
Please never try to emit WARNING: or BUG: message just for letting developers update their code. If you can't use panic() instead of WARN*() or BUG*(), you should not use WARN*() or BUG*(). The patch author who changes the behavior of in-tree code is responsible for updating all in-tree code.
Commit 860e748bddcc ("drm: ensure blend mode supported if pixel format with alpha exposed") must be reverted or updated not to emit "WARNING:" string. Until authors of that commit respond, that commit had better be excluded from the linux-next tree.
Quoted 4 lines · Tetsuo Handa
Quoted 4 lines
i.e. I was wondering whether the system could ignore the "WARNING:"s (and avoid the kernel panicking on them etc.) until after boot, i.e. when the fuzzing starts.
It could, but it would involve making a change in syzbot. Specifically, syzbot would have to be changed to boot the kernel being tested without the kernel.panic_on_warn=1 boot command line function, and then in the syzbot reproducer, to add the moral equivalent of
echo 1 > /proc/sys/kernel/panic_on_warn
Quoted 4 lines
Commit
860e748bddcc("drm: ensure blend mode supported if pixel format with alpha exposed") must be reverted or updated not to emit "WARNING:" string. Until authors of that commit respond, that commit had better be excluded from the linux-next tree.
There appears to be a fix which has been reviewed; I found it by searching lore.kernel.org for the string "drm: ensure blend mode":
https://lore.kernel.org/all/CAPM=9txcOcnnxznq41z8+FA_V2BCqmRKFZBOt0oEhcHbBs8Fsg@mail.gmail.com/
- Ted
Quoted 7 lines · Theodore Tso
It could, but it would involve making a change in syzbot. Specifically, syzbot would have to be changed to boot the kernel being tested without the kernel.panic_on_warn=1 boot command line function, and then in the syzbot reproducer, to add the moral equivalent of
echo 1 >
/proc/sys/kernel/panic_on_warn
Thanks Ted! Yeah, that's about what I was expecting.
Cheers,
Miguel
Quoted 6 lines · Theodore Tso
Quoted 4 lines · Tetsuo Handa
Quoted 4 lines
i.e. I was wondering whether the system could ignore the "WARNING:"s (and avoid the kernel panicking on them etc.) until after boot, i.e. when the fuzzing starts.
Quoted 6 lines
It could, but it would involve making a change in syzbot. Specifically, syzbot would have to be changed to boot the kernel being tested without the kernel.panic_on_warn=1 boot command line function, and then in the syzbot reproducer, to add the moral equivalent of
echo 1 >
/proc/sys/kernel/panic_on_warn
That does not help. syzbot stops not because the kernel calls panic() but because the kernel emits "WARNING:" etc. followed by a stack trace. The list of patterns is at
https://github.com/google/syzkaller/blob/master/pkg/report/linux.go .
For example, loop_check_io_race() in
https://lkml.kernel.org/r/[email protected]
will be sufficient to stop syzbot. Mimicking BUG*() or WARN*() using printk() + stack dump function in order to get a difficult to reproduce problem reported by syzbot is helpful. But unfortunately, no response to my patch for two weeks... :-(
Quoted 5 lines · Tetsuo Handa
That does not help. syzbot stops not because the kernel calls
panic()but because the kernel emits "WARNING:" etc. followed by a stack trace. The list of patterns is at
https://github.com/google/syzkaller/blob/master/pkg/report/linux.go .
The point is that syzbot could skip those until a given point at boot. But in order to do that, of course, you still need to disable the panicking.
Cheers,
Miguel
Quoted 11 lines · Miguel Ojeda
Quoted 5 lines · Tetsuo Handa
That does not help. syzbot stops not because the kernel calls
panic()but because the kernel emits "WARNING:" etc. followed by a stack trace. The list of patterns is at
https://github.com/google/syzkaller/blob/master/pkg/report/linux.go .The point is that syzbot could skip those until a given point at boot. But in order to do that, of course, you still need to disable the panicking.
Suppose syzbot defers evaluation of
------------[ cut here ]------------ [PLANE:35:plane-0] pixel format with alpha exposed but blend mode not setup WARNING: drivers/gpu/drm/drm_mode_config.c:873 at validate_blend_mode_for_alpha_formats drivers/gpu/drm/drm_mode_config.c:872 [inline], CPU#1: swapper/0/1 WARNING: drivers/gpu/drm/drm_mode_config.c:873 at drm_mode_config_validate+0x1c6a/0x1e60 drivers/gpu/drm/drm_mode_config.c:938, CPU#1: swapper/0/1 (...snipped...)
messages until a given point at boot, how can syzbot determine whether it is appropriate to start (or continue) fuzz testing? This specific WARNING: is harmless but almost all other WARNING: indicates something is wrong (e.g. the kernel's state might be no longer consistent), like panic_on_warn is there. Deferring could mitigate "being unable to follow daily next-YYYYMMDD" problem, but it can force syzbot to fuzz-test on already broken kernels (adds a new problem).
If a Gen-AI is used while running syzkaller and that AI is capable of evaluating whether it is appropriate to start (or continue) fuzz testing when encountering individual WARNING: messages, syzkaller could skip this specific WARNING: as harmless. But unlike Sashiko, syzkaller itself is not meant to require GPU for running AI models when the kernel to be fuzz-tested is running.
I feel that the discussion is getting out of the point. Like I explained at
https://lkml.kernel.org/r/[email protected] , the point of this discussion is how to address the technical problem and the procedural problem. For the former, I will start a new thread. But for the latter, I sent "[PATCH v5] loop: Fix NULL pointer dereference in lo_rw_aio()" to ML but got no responses / interests from developers / maintainers. This bug remains undebuggable if you don't want me to send this patch to linux-next tree via my tree while saying "Not my business.".
Quoted 6 lines · Tetsuo Handa
syzbot uses kernel.panic_on_warn=1, for most of WARN*() users are reporting problems that should not be ignored. Therefore, WARN*() must not be used for reporting recoverable mistakes like
-EINVAL. Abusing WARN*() like
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/kernel/workqueue.c?id=6e845bcb78c95af935094040bd4edc3c2b6dd784#n5855
is a fatal barrier for continuous testing systems like syzbot.
Yeah, a lot of CIs that grep for certain keywords in the kernel log (or that simply enable the panicking) suffer from those, but the idea I was mentioning is to defer that until the fuzzing starts -- see Ted's reply, it sounds potentially doable.
It may not solve the complete problem, but it may allow you, for certain trees or periods of time, to continue your testing. Thus removing a bit of the coupling between you and other maintainers, i.e. making life easier for both sides.
I don't know -- I am just trying to find compromises that could help you.
Cheers,
Miguel
Quoted 29 lines · Tetsuo Handa
Quoted 11 lines · Miguel Ojeda
Quoted 2 lines · Tetsuo Handa
syzbot stops upon encountering "BUG:" or "WARNING:" within
printk()output.Out of curiosity, is that a fundamental property or something that could be optionally be relaxed (for certain linux-next runs only, perhaps) so that you can continue testing in certain cases where the "WARNING:" may be harmless (i.e. irrelevant for your particular fuzzing)?
Quoted 3 lines
i.e. I was wondering whether the system could ignore the "WARNING:"s (and avoid the kernel panicking on them etc.) until after boot, i.e. when the fuzzing starts.
syzbot uses kernel.panic_on_warn=1, for most of WARN*() users are reporting problems that should not be ignored. Therefore, WARN*() must not be used for reporting recoverable mistakes like
-EINVAL. Abusing WARN*() like
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/kernel/workqueue.c?id=6e845bcb78c95af935094040bd4edc3c2b6dd784#n5855
is a fatal barrier for continuous testing systems like syzbot.Quoting from https://lkml.kernel.org/r/[email protected] :
Please never try to emit WARNING: or BUG: message just for letting developers update their code. If you can't use
panic()instead of WARN*() or BUG*(), you should not use WARN*() or BUG*(). The patch author who changes the behavior of in-tree code is responsible for updating all in-tree code.
Quoted 5 lines
Commit
860e748bddcc("drm: ensure blend mode supported if pixel format with alpha exposed") must be reverted or updated not to emit "WARNING:" string. Until authors of that commit respond, that commit had better be excluded from the linux-next tree.
Hello,
First of all, sorry for all the trouble with this warning. I'll propose a patch replacing the warning with printk() or pr_info(), if that doesn't cause similar issues for fuzzers.
Thanks,
Leandro
PATCH 1/1 lockdep: Enable the printing of held locks of running non-current tasks (was: Policy regarding linux-next only changes)
In reply to Tetsuo HandaQuoted 6 lines · Tetsuo Handa
Quoted 5 lines · Boqun Feng
Quoted 5 lines
Quoted 5 lines
While testing lockdep changes on linux-next, I found there is a commit
ca65ccfdc5b3("locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.
Yeah, so while it's still true that printing out their held locks array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
The (lightly tested) patch below implements this.
Note that this way there's no need to gate this on the Syzkaller config option, and you wouldn't have to carry it in -next either, it's a useful mainline kernel debuggability enhancement in its own right.
Would this work for you?
Thanks,
Ingo
=================>
From: Ingo Molnar <[email protected]>
Date: Sun, 5 Jul 2026 10:38:06 +0200
Subject: [PATCH] lockdep: Enable the printing of held locks of running non-current tasks
Currently lockdep does not print out the held locks of non-current tasks that are running (on some other CPU), due to the fact that the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking bugs is easier if we print this information too, because the more locking information the merrier. In particular races are bound to have multiple tasks running on different CPUs, and the exclusion of their held locks information is unnecessarily limiting.
So while it's still true that printing out their held locks array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
This commit implements that change.
For running tasks, the printout adds a warning about the fact that the task is running and that the held locks array is in flux:
5 locks held by bash/1234 (WARNING: task running):
...
Note that while re-flowing the function this change also micro-optimizes the common !depth case, which unnecessarily ran through the runnability check and the (zero-steps) loop.
--- kernel/locking/lockdep.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
kernel/locking/lockdep.c +12 -7
@@ -786,18 +786,23 @@ static void print_lock(struct held_lock *hlock)static void lockdep_print_held_locks (struct task_struct *p) {int i, depth =READ_ONCE (p->lockdep_depth); +const char *msg_running ="" ; -if (!depth) +if (!depth) {printk ("no locks held by %s/%d.\n" , p->comm,task_pid_nr (p)); -else -printk ("%d lock%s held by %s/%d:\n" , depth, -str_plural (depth), p->comm,task_pid_nr (p)); +return ; + } +/* -* It's not reliable to print a task's held locks if it's not sleeping -* and it's not the current task. +* It's not reliable to print a task's held locks if it's not +* sleeping and it's not the current task: */ if (p != current &&task_is_running (p)) -return ; + msg_running =" (WARNING: task running)" ; + +printk ("%d lock%s held by %s/%d%s:\n" , depth, +str_plural (depth), p->comm,task_pid_nr (p), msg_running); +for (i =0 ; i < depth; i++) {printk (" #%d: " , i);print_lock (p->held_locks + i);
Re: [PATCH] lockdep: Enable the printing of held locks of running non-current tasks
Quoted 4 lines · Ingo Molnar
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Quoted 8 lines
The (lightly tested) patch below implements this.
Note that this way there's no need to gate this on the Syzkaller config option, and you wouldn't have to carry it in -next either, it's a useful mainline kernel debuggability enhancement in its own right.
Would this work for you?
Yes, this will be a helpful change. Thank you very much.
I can drop "locking/lockdep: make lockdep_print_held_locks() always print if CONFIG_DEBUG_AID_FOR_SYZBOT=y" change from linux-next tree.
Please avoid emitting "BUG:" or "WARNING:", for syzkaller stops upon encountering these strings. Also, since printk() is a slow operation, please reduce number of characters to print where reasonable.
- msg_running = " (WARNING: task running)";
+ msg_running = " (running)";
If we can safely calculate on which CPU that thread is running (or waiting to run), printing CPU number might be also helpful.
char msg_running[14] = "";
if (task_is_running(p)) {
int cpu_id = which_cpu_task_is_on(p); // If possible...
if (cpu_id >= 0)
scnprintf(msg_running, sizeof(msg_running), "(C%u)", cpu_id);
else
scnprintf(msg_running, sizeof(msg_running), "(running)");
}Ingo Molnar
AppliedPATCH 1/1 lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Quoted 28 lines · Tetsuo Handa
Quoted 4 lines · Ingo Molnar
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Quoted 8 lines
The (lightly tested) patch below implements this.
Note that this way there's no need to gate this on the Syzkaller config option, and you wouldn't have to carry it in -next either, it's a useful mainline kernel debuggability enhancement in its own right.
Would this work for you?
Yes, this will be a helpful change. Thank you very much.
I can drop "locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y" change from linux-next tree.Please avoid emitting "BUG:" or "WARNING:", for syzkaller stops upon encountering these strings. Also, since
printk()is a slow operation, please reduce number of characters to print where reasonable.(quoted diff) +1
-1- msg_running = " (WARNING: task running)"; + msg_running = " (running)";If we can safely calculate on which CPU that thread is running (or waiting to run), printing CPU number might be also helpful.
Quoted 6 lines
char msg_running[14] = "";
if (task_is_running(p)) {
int cpu_id = which_cpu_task_is_on(p); // If possible...
if (cpu_id >= 0)[ Side note: task_cpu() is never negative and can always be relied on
for valid tasks to be an int in the possible-CPUs numeric range. ]
Quoted 4 lines
scnprintf(msg_running, sizeof(msg_running), "(C%u)", cpu_id); else scnprintf(msg_running, sizeof(msg_running), "(running)"); }
Sure, we can print the CPU ID too, in fact we can do something even
better: for locking races it's useful information on which CPU a task last ran on, right? So I think we should just print the CPU ID of tasks unconditionally.
The patch below does this, and also streamlines the printout a bit, into a single statement. The new message variants are:
[ 37.078569] locks held by sleep/7991: 1, on CPU#2: [ 37.083565] locks held by sleep/7995: 1, on CPU#3: [ 37.086346] locks held by sleep/7997: 5, last CPU#2: [ 37.096518] locks held by sleep/8001: 6, last CPU#0 [ 37.056812] locks held by bash/414: 0, last CPU#0
See the patch description and the extended comment in the code about the details.
I've applied this patch to tip:locking/debug, so it should show up in -next tomorrow, but it can be iterated some more.
Thanks,
Ingo
================================>
From: Ingo Molnar <[email protected]>
Date: Sun, 5 Jul 2026 11:05:17 +0200
Subject: [PATCH] lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Background:
==========
Currently lockdep does not print out the held locks of non-current tasks that are running on some other CPU, due to the fact that the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking bugs is easier if we print this information too, because the more locking information the merrier. In particular races are bound to have multiple tasks running on different CPUs, and the exclusion of their held locks information is unnecessarily limiting.
So while it's still true that printing out their held locks array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print.
--- kernel/locking/lockdep.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
kernel/locking/lockdep.c +25 -9
@@ -787,17 +787,33 @@ static void lockdep_print_held_locks(struct task_struct *p) {int i, depth =READ_ONCE (p->lockdep_depth); -if (!depth) -printk ("no locks held by %s/%d.\n" , p->comm,task_pid_nr (p)); -else -printk ("%d lock%s held by %s/%d:\n" , depth, -str_plural (depth), p->comm,task_pid_nr (p));/* -* It's not reliable to print a task's held locks if it's not sleeping -* and it's not the current task. +* Note that it's always somewhat unreliable to print held locks +* of a task that is running on another CPU, but we cannot guarantee +* the stability of ->held_locks without actually stopping all active +* remote CPUs, which we absolutely do not want to do because it's +* very intrusive and thus slow. +* +* So we do the next best thing here: we print out the held lock +* array on a best-effort basis, without crashing even if the +* fields are being modified on another CPU. Note the careful +* construction of print_lock() so that it never crashes. +* +* We also print out the CPU the task is or was last running on, with +* the message saying 'on CPU...' if the task is running, and +* 'last CPU' if it's not. +* +* Also note that the task_is_running(p) information is fundamentally +* racy: even if the message says the task is 'on CPU', the task may +* have scheduled out already, or if it says 'last CPU', it may just +* have scheduled in on another CPU. But even with these limitations +* it's still useful debuggining information. */ -if (p != current &&task_is_running (p)) -return ; +printk ("locks held by %s/%d: %d, %s CPU#%d%s\n" , + p->comm,task_pid_nr (p), depth, +task_is_running (p) ?"last" :"on" ,task_cpu (p), + depth >0 ?":" :"" ); +for (i =0 ; i < depth; i++) {printk (" #%d: " , i);print_lock (p->held_locks + i);
Re: [PATCH -v2] lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Quoted 3 lines · Ingo Molnar
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Is there a warning we print out?
We dropped WARNING: in this version, didn't we?
Quoted 13 lines
Implementation:
==============Implement that change.
Also re-flow the function and streamline the printout into a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a dependency on English spelling of plurals, to a uniform:locks held by bash/1234: %d
Quoted 7 lines
Which spells correctly for 0, 1 and higher values, and should also be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print.
I like this change. Thank you. ;-)
# echo t > /proc/sysrq-trigger; sleep 3; dmesg -c | cut -b 15-
Before:
Showing all locks held in the system:
4 locks held by pr/legacy/16:
1 lock held by rcub/1/18:
#0: ffffffff92ba36c8 (rcu_state){+.+.}-{0:0}, at: rcu_boost_kthread+0x15c/0x5d0
3 locks held by bash/1221:
#0: ffff8cd5dd8df500 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x9a/0x150
#1: ffffffff92b9e080 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x56/0x140
#2: ffffffff92b9e080 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x3d/0x184After:
Showing all locks held in the system:
locks held by pr/legacy/16: 4, last CPU#0:
#0: ffffffffaaf9a4a0 (console_lock){+.+.}-{0:0}, at: legacy_kthread_func+0x85/0x190
#1: ffffffffaaf9a4f8 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x92/0x710
#2: ffffffffab1fb818 (printing_lock){+.+.}-{3:3}, at: vt_console_print+0x74/0x640
#3: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: rt_spin_trylock+0x61/0x140
locks held by rcub/1/18: 1, on CPU#0:
#0: ffffffffaafa36c8 (rcu_state){+.+.}-{0:0}, at: rcu_boost_kthread+0x15c/0x5d0
locks held by bash/1206: 3, last CPU#0:
#0: ffff8c62894da500 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x9a/0x150
#1: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x56/0x140
#2: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x3d/0x184Re: [PATCH -v2] lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Quoted 7 lines · Tetsuo Handa
Quoted 3 lines · Ingo Molnar
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Is there a warning we print out?
We dropped WARNING: in this version, didn't we?
Indeed - I fixed the changelog.
Quoted 22 lines
Quoted 4 lines
Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print.
I like this change. Thank you. ;-)
# echo t > /proc/sysrq-trigger; sleep 3; dmesg -c | cut -b 15-Before:
Showing all locks held in the system: 4 locks held by pr/legacy/16: 1 lock held by rcub/1/18: #0: ffffffff92ba36c8 (rcu_state){+.+.}-{0:0}, at: rcu_boost_kthread+0x15c/0x5d0 3 locks held by bash/1221: #0: ffff8cd5dd8df500 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x9a/0x150 #1: ffffffff92b9e080 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x56/0x140 #2: ffffffff92b9e080 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x3d/0x184After:
Quoted 12 lines
Showing all locks held in the system:
locks held by pr/legacy/16: 4, last CPU#0:
#0: ffffffffaaf9a4a0 (console_lock){+.+.}-{0:0}, at: legacy_kthread_func+0x85/0x190
#1: ffffffffaaf9a4f8 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x92/0x710
#2: ffffffffab1fb818 (printing_lock){+.+.}-{3:3}, at: vt_console_print+0x74/0x640
#3: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: rt_spin_trylock+0x61/0x140
locks held by rcub/1/18: 1, on CPU#0:
#0: ffffffffaafa36c8 (rcu_state){+.+.}-{0:0}, at: rcu_boost_kthread+0x15c/0x5d0
locks held by bash/1206: 3, last CPU#0:
#0: ffff8c62894da500 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x9a/0x150
#1: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x56/0x140
#2: ffffffffaaf9e080 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x3d/0x184Great, and you are welcome.
I've added your Tested-by tag as well, and graduated the updated commit to tip:locking/core, to be merged into v7.3 if everything goes fine.
Thanks,
Ingo
[tip: locking/debug] lockdep: Enable the printing of held locks of remote running tasks and print task CPU
In reply to Ingo MolnarThe following commit has been merged into the locking/debug branch of tip:
The following commit has been merged into the locking/debug branch of tip:
Commit-ID: 80b445aaa7e2ef4d62afc7857a955a4b60c40565 Gitweb: https://git.kernel.org/tip/80b445aaa7e2ef4d62afc7857a955a4b60c40565 Author: Ingo Molnar <[email protected]> AuthorDate: Sun, 05 Jul 2026 11:05:17 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 07 Jul 2026 09:07:21 +02:00
lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Background:
==========
Currently lockdep does not print out the held locks of non-current tasks that are running on some other CPU, due to the fact that the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking bugs is easier if we print this information too, because the more locking information the merrier. In particular races are bound to have multiple tasks running on different CPUs, and the exclusion of their held locks information is unnecessarily limiting.
So while it's still true that printing out their held locks array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out a warning when we print such a task, so that users are aware of any racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print.
--- kernel/locking/lockdep.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
kernel/locking/lockdep.c +25 -9
@@ -787,17 +787,33 @@ static void lockdep_print_held_locks(struct task_struct *p) {int i, depth =READ_ONCE (p->lockdep_depth); -if (!depth) -printk ("no locks held by %s/%d.\n" , p->comm,task_pid_nr (p)); -else -printk ("%d lock%s held by %s/%d:\n" , depth, -str_plural (depth), p->comm,task_pid_nr (p));/* -* It's not reliable to print a task's held locks if it's not sleeping -* and it's not the current task. +* Note that it's always somewhat unreliable to print held locks +* of a task that is running on another CPU, but we cannot guarantee +* the stability of ->held_locks without actually stopping all active +* remote CPUs, which we absolutely do not want to do because it's +* very intrusive and thus slow. +* +* So we do the next best thing here: we print out the held lock +* array on a best-effort basis, without crashing even if the +* fields are being modified on another CPU. Note the careful +* construction of print_lock() so that it never crashes. +* +* We also print out the CPU the task is or was last running on, with +* the message saying 'on CPU...' if the task is running, and +* 'last CPU' if it's not. +* +* Also note that the task_is_running(p) information is fundamentally +* racy: even if the message says the task is 'on CPU', the task may +* have scheduled out already, or if it says 'last CPU', it may just +* have scheduled in on another CPU. But even with these limitations +* it's still useful debuggining information. */ -if (p != current &&task_is_running (p)) -return ; +printk ("locks held by %s/%d: %d, %s CPU#%d%s\n" , + p->comm,task_pid_nr (p), depth, +task_is_running (p) ?"last" :"on" ,task_cpu (p), + depth >0 ?":" :"" ); +for (i =0 ; i < depth; i++) {printk (" #%d: " , i);print_lock (p->held_locks + i);
[tip: locking/core] lockdep: Enable the printing of held locks of remote running tasks and print task CPU
In reply to Ingo MolnarThe following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 051f5d223dfc1806e216f60e4b29c1bf35f5c2d3 Gitweb: https://git.kernel.org/tip/051f5d223dfc1806e216f60e4b29c1bf35f5c2d3 Author: Ingo Molnar <[email protected]> AuthorDate: Sun, 05 Jul 2026 11:05:17 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 08 Jul 2026 10:36:23 +02:00
lockdep: Enable the printing of held locks of remote running tasks and print task CPU
Background:
==========
Currently lockdep does not print out the held locks of non-current tasks that are running on some other CPU, due to the fact that the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking bugs is easier if we print this information too, because the more locking information the merrier. In particular races are bound to have multiple tasks running on different CPUs, and the exclusion of their held locks information is unnecessarily limiting.
So while it's still true that printing out their held locks array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out the information as well that a task is running, so that users are aware of any racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print.
--- kernel/locking/lockdep.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
kernel/locking/lockdep.c +25 -9
@@ -787,17 +787,33 @@ static void lockdep_print_held_locks(struct task_struct *p) {int i, depth =READ_ONCE (p->lockdep_depth); -if (!depth) -printk ("no locks held by %s/%d.\n" , p->comm,task_pid_nr (p)); -else -printk ("%d lock%s held by %s/%d:\n" , depth, -str_plural (depth), p->comm,task_pid_nr (p));/* -* It's not reliable to print a task's held locks if it's not sleeping -* and it's not the current task. +* Note that it's always somewhat unreliable to print held locks +* of a task that is running on another CPU, but we cannot guarantee +* the stability of ->held_locks without actually stopping all active +* remote CPUs, which we absolutely do not want to do because it's +* very intrusive and thus slow. +* +* So we do the next best thing here: we print out the held lock +* array on a best-effort basis, without crashing even if the +* fields are being modified on another CPU. Note the careful +* construction of print_lock() so that it never crashes. +* +* We also print out the CPU the task is or was last running on, with +* the message saying 'on CPU...' if the task is running, and +* 'last CPU' if it's not. +* +* Also note that the task_is_running(p) information is fundamentally +* racy: even if the message says the task is 'on CPU', the task may +* have scheduled out already, or if it says 'last CPU', it may just +* have scheduled in on another CPU. But even with these limitations +* it's still useful debuggining information. */ -if (p != current &&task_is_running (p)) -return ; +printk ("locks held by %s/%d: %d, %s CPU#%d%s\n" , + p->comm,task_pid_nr (p), depth, +task_is_running (p) ?"last" :"on" ,task_cpu (p), + depth >0 ?":" :"" ); +for (i =0 ; i < depth; i++) {printk (" #%d: " , i);print_lock (p->held_locks + i);
Boqun Feng
maintainerQuoted 16 lines · Tetsuo Handa
Quoted 15 lines · Boqun Feng
Quoted 14 lines
Quoted 5 lines
While testing lockdep changes on linux-next, I found there is a commit
ca65ccfdc5b3("locking/lockdep: makelockdep_print_held_locks()always print ifCONFIG_DEBUG_AID_FOR_SYZBOT=y") applied to lockdep. Checking the history it looks like this commit has been in linux-next for quite a while (at least 2 months); there were no emails on LKML about this commit at all.Yes, these patches are for debugging difficult problems.
For example,
ca65ccfdc5b3has been a concern for almost 8 years without progress because developers show little interest for this change:https://lkml.kernel.org/r/1535975097-19080-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
https://lkml.kernel.org/r/[email protected]
Quoted 8 lines
Quoted 3 lines
This email was from 6 years ago, if that's the latest version, I'm not so sure you could say "developers show little interest for this change". Maybe you could provide more context about this that I missed?
I don't remember whether some discussions are done somewhere else. But judging from the fact that code remains unchanged, nobody else has attempted or succeeded to make this change.
Maybe we have some misunderstanding here, I'm not following you. Nobody else has attempted or succeeded to make the change maybe because the change is not important to anyone else. So if you want to make that chagne, you are kinda responsible to keep the communication going. If the communication stopped since 6 years ago, I cannot think of anyone better than you should bring back the communication, but silently putting it in linux-next is certainly not the right way. Since you stopped reposting the patch 6 years ago, people may think your problem has been resolved in other ways.
Yes, it can be frustrating sometimes (maybe always :)), when you have to keep remindering people about an issue, but this is the nature of the human being communication. People ignore conversations for mutliple
reasons: too busy, losing interests, getting interrupted, wanting to get a proper response but getting interrupted, etc. So if you think the topic is important, we are looking forwards to your help to drive the communication. Thank you!
[...]
Quoted 6 lines
Quoted 4 lines
By "silently" I mean I didn't see you gave a heads-up about doing this either, e.g. an email saying "hey I'm including these into the linux-next". Maybe I'm missing something here?
I'm fine with announcing to ML that I am about to make debug/temporary changes.
You should have in this case, until Gary raised this up, it was poorly communicatd (because there is no intent to communicate).
Regards,
Boqun
Quoted 4 lines
But we might want a prefix for debug/temporary patches that are not intended for upstream? Also, we might want a kernel config option (e.g.
CONFIG_DEBUG_AID_FOR_SYZBOT) for minimizing users affected?
Mark Brown
maintainerQuoted 11 lines · Gary Guo
For obvious reasons I have concern about this practice; being a maintainer shouldn't mean that you're allowed to add random stuff to linux-next. This reminds me of another case last month
https://lore.kernel.org/all/[email protected]/ where the added stuff via TOMOYO tree breaks Rust build on linux-next. I think this practice is questionable and I would like a clarification on the policy of adding stuff that is not upstream material.I recognize that there is a need to add commits that serve as testing and debugging purpose to linux-next, but I think any of these changes should be temporary in nature and have acknowledgement from maintainers of touched code.
Yes, the idea is that -next should normally just have things that are currently intended to go to Linus by the end of the next merge window. Obviously there's some flexability, especially for things that are within the tree itself, but we're trying to get a heads up on things we'll see in mainline not just do random experimental stuff. I'd especially not to see people putting in changes specific to other trees without some coordination with the maintainers of the other trees.
Quoted 2 lines
Perhaps these changes should also be in a separate topic branch which gets pulled from linux-next temporarily.
People do sometimes ask to put separate trees in for things like this, it does depend a bit on how the branch that goes into -next is constructed. Some trees put a merge of various branches into -next but send the individual branches to Linus for example, they can just add something to their merge easily enough.