Skip to contentSkip to conversation

Squashing bugs!

Ksummit-discuss Shuah Khan 11 participants 31 messages May 30, 2019 – Jun 5, 2019

lore
Share & tools
lore

Shuah Khan

author
#0 lore raw

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

#1 lore raw
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Thanks,
Laura

author
#2 lore raw
Quoted 30 lines · Laura Abbott
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Quoted 8 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

I am hoping we can look at all bugs as well. Syzbot happens to be just a good starting for me to start this discussion.

thanks,
-- Shuah

#3 lore raw
Quoted 40 lines · Shuah Khan
Quoted 30 lines · Laura Abbott
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Quoted 8 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Quoted 2 lines

I am hoping we can look at all bugs as well. Syzbot happens to be just a good starting for me to start this discussion.

I would say that single place to see/report/manage all bugs (it can be per-subsystem or for whole kernel) and ability to set priority will do the trick.

Maybe, it is worth to integrate syzbot dashboard into kernel bugzilla, at least for priority.

Thanks

Quoted 8 lines

thanks,
-- Shuah

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#4 lore raw
In reply to Laura Abbott
Quoted 9 lines · Laura Abbott
Quoted 7 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

Many seem to get discussed for a while with people all mostly agreeing on what needs to be done, but then nothing actually materializes.

Now, part of this, I think, is a lack of an "objective" sense of priority for the various bugs on the dashboard. Their frequency of appearance (currently the only thing that really turns red), does not reflect anything about their severity nor reachability. Severity is pretty straight forward: stack smashing, use-after-free, etc are way worse (generally speaking) than crash/hang or WARN(). For reachability, if we could narrow the scope of things a bit more ("this is reachable only by init namespace root", or "only reachable via CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any modern distro .config") that would be great.

Quoted 13 lines
Quoted 13 lines

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 4 lines
Quoted 4 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

What I want to find is a way to fund on-going work in this area. It just does not appear to be working to find funding within various companies to do this kind of sustained bug-hunting. It's a bit of a resource conflict: if a company has kernel developers on staff they tend to want to have them focused on "new things".

Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

I'd especially love to see deduplication and cross-referencing working a little more automatically between trackers. "Oh, I've seen that before over here *link*" by a machine would be quite helpful.

Signature · 1 line
Kees Cook
#5 lore raw
Quoted 14 lines · Kees Cook
Quoted 9 lines · Laura Abbott
Quoted 7 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

Many seem to get discussed for a while with people all mostly agreeing on what needs to be done, but then nothing actually materializes.

Quoted 10 lines

Now, part of this, I think, is a lack of an "objective" sense of priority for the various bugs on the dashboard. Their frequency of appearance (currently the only thing that really turns red), does not reflect anything about their severity nor reachability. Severity is pretty straight forward: stack smashing, use-after-free, etc are way worse (generally speaking) than crash/hang or WARN(). For reachability, if we could narrow the scope of things a bit more ("this is reachable only by init namespace root", or "only reachable via CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any modern distro .config") that would be great.

Yes, I think this would be really useful.

Quoted 18 lines
Quoted 13 lines
Quoted 13 lines

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 4 lines
Quoted 4 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

Quoted 5 lines

What I want to find is a way to fund on-going work in this area. It just does not appear to be working to find funding within various companies to do this kind of sustained bug-hunting. It's a bit of a resource conflict: if a company has kernel developers on staff they tend to want to have them focused on "new things".

Well, not necessarily only "new things". Distros have kernel people that do bugfixing but the most focus naturally goes towards bugs customers are likely to get exposed to. Triaging through syzbot reports when you have enough bugs your customers actually hit thus does not look very appealing :) But this goes back to your idea of automated way of classifying bug priority. Unpriviledged user triggerable kernel oops in MM is going to get more attention than init namespace root triggerable one in some random old driver (which is likely to bitrot for long and that's just a reality of limited resources)...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#6 lore raw
Quoted 5 lines · Jan Kara
Quoted 5 lines

What I want to find is a way to fund on-going work in this area. It just does not appear to be working to find funding within various companies to do this kind of sustained bug-hunting. It's a bit of a resource conflict: if a company has kernel developers on staff they tend to want to have them focused on "new things".

Quoted 9 lines

Well, not necessarily only "new things". Distros have kernel people that do bugfixing but the most focus naturally goes towards bugs customers are likely to get exposed to. Triaging through syzbot reports when you have enough bugs your customers actually hit thus does not look very appealing :) But this goes back to your idea of automated way of classifying bug priority. Unpriviledged user triggerable kernel oops in MM is going to get more attention than init namespace root triggerable one in some random old driver (which is likely to bitrot for long and that's just a reality of limited resources)...

Part of it is also a matter of how much time you want to spend. There are some bugs I can spend 30 minutes looking at and know I can either come up with the proper fix or write up a report that's likely to get fixed by a maintainer. Then there are others which look like they could be solvable with more time that I don't have. The long tail of low priority bugs are the ones that would be really good candidates for interested community kernel developers to spend some time on.

I'll also just state straight out that just doing bug triage gets very tedious after a while. It's valuable work but after a while you aren't actually growing your skills very much. This might be something that would be best as a yearly contract position. Of course, people who aren't me might also have more patience for it long term :)

Thanks,
Laura

author
#7 lore raw
In reply to Kees Cook
Quoted 8 lines · Kees Cook
Quoted 7 lines · Laura Abbott
Quoted 5 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

Many seem to get discussed for a while with people all mostly agreeing on what needs to be done, but then nothing actually materializes.

Right. My experience with a fix I sent is that there is no agreement on how to fix between maintainers. Hence, it is in limbo.

Quoted 11 lines

Now, part of this, I think, is a lack of an "objective" sense of priority for the various bugs on the dashboard. Their frequency of appearance (currently the only thing that really turns red), does not reflect anything about their severity nor reachability. Severity is pretty straight forward: stack smashing, use-after-free, etc are way worse (generally speaking) than crash/hang or WARN(). For reachability, if we could narrow the scope of things a bit more ("this is reachable only by init namespace root", or "only reachable via CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any modern distro .config") that would be great.

Agree. It is very hard to parse the information to determine priority. I have been looking at how often it can be reproduced. In some cases, it isn't clear how likely for this bug to be triggered in regular cases. This is where syzbot adds value that it stresses error cases.

This one bug I am looking at is in an error leg, which is nice. However, syzbot reports are hard to manage. I found 3 instances of the same bug, with fix pulled into the mainline associated with one of the 3.

I spent time on Friday to find a quick way to reply to one of them with list of IDs. It will be great to have a ID associated with these bugs, so I can go say, hey this fix applies to these IDs.

Quoted 18 lines
Quoted 13 lines
Quoted 13 lines

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 4 lines
Quoted 4 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

Quoted 6 lines

What I want to find is a way to fund on-going work in this area. It just does not appear to be working to find funding within various companies to do this kind of sustained bug-hunting. It's a bit of a resource conflict: if a company has kernel developers on staff they tend to want to have them focused on "new things".

Yes. This is my concern as well. This is why I am trying to see if we can find resources. I would like to start with trying to channel the resources from the mentorship program for this. I do know that this adds overhead as well. Maybe some low hanging fruit and easy fix type bugs can be taken care of using this program and then we can focus our expert energies on harder bugs.

Quoted 8 lines
Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Yes. This is what I am finding. Syzbot or any other bug tracking system to be effective, it has to be easy to manage and mark bugs as duplicate, closed, fixed. etc.

thanks,
-- Shuah

#8 lore raw
In reply to Laura Abbott
Quoted 30 lines · Laura Abbott
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Maybe the solution here is to standardize bug reporting bots? A [email protected] mailing list which our various bots could report to with a standard format that'll allow users/maintainers to easily filter it to get only bugs they care about?

I'd also love to get a concrete solution we could actually go ahead and try for a year. It doesn't have to be pretty or perfect, but just something we could experiment with and possibly build upon later? We keep kicking this can down the road for way too long.

--
Thanks,
Sasha

author
#9 lore raw
Quoted 39 lines · Sasha Levin
Quoted 30 lines · Laura Abbott
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Quoted 5 lines

Maybe the solution here is to standardize bug reporting bots? A [email protected] mailing list which our various bots could report to with a standard format that'll allow users/maintainers to easily filter it to get only bugs they care about?

Easy to filter and manage is important for usability. I mentioned in my response Kees that I am finding it hard to update syzbot bug status to mark them as duplicate and/or fixed. The email interface is nice for individual bugs, but not very easy when it comes to other actions.

Quoted 5 lines

I'd also love to get a concrete solution we could actually go ahead and try for a year. It doesn't have to be pretty or perfect, but just something we could experiment with and possibly build upon later? We keep kicking this can down the road for way too long.

I am hoping we will come up something concrete.

thanks,
-- Shuah

#10 lore raw
Quoted 14 lines · Shuah Khan
Quoted 8 lines
Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Quoted 5 lines

Maybe the solution here is to standardize bug reporting bots? A [email protected] mailing list which our various bots could report to with a standard format that'll allow users/maintainers to easily filter it to get only bugs they care about?

Quoted 4 lines

Easy to filter and manage is important for usability. I mentioned in my response Kees that I am finding it hard to update syzbot bug status to mark them as duplicate and/or fixed. The email interface is nice for individual bugs, but not very easy when it comes to other actions.

May I recommend using a project like git-bug [1] instead of a mailing list? I am increasingly worried that with mail services being increasingly centralized across 4-5 major providers we'll soon find it quite difficult to continue using mailing lists and have them reliably deliver mail to all participants. Heck, I've been battling with Gmail over the weekend due to it arbitrarily deciding that Robert Richter is receiving way too much mail.

Git-bug, on the other hand, is a great internal bug tracker for automatically generated bug reports -- it already provides the necessary structured platform, is decentralized, easily clonable and offline-able, and does not suffer from some mail provider arbitrarily deciding throttle its traffic for no obvious reason.

.. [1] https://github.com/MichaelMure/git-bug

-K

author
#11 lore raw
Quoted 20 lines · Konstantin Ryabitsev
Quoted 14 lines · Shuah Khan
Quoted 8 lines
Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Quoted 5 lines

Maybe the solution here is to standardize bug reporting bots? A [email protected] mailing list which our various bots could report to with a standard format that'll allow users/maintainers to easily filter it to get only bugs they care about?

Quoted 4 lines

Easy to filter and manage is important for usability. I mentioned in my response Kees that I am finding it hard to update syzbot bug status to mark them as duplicate and/or fixed. The email interface is nice for individual bugs, but not very easy when it comes to other actions.

Quoted 6 lines

May I recommend using a project like git-bug [1] instead of a mailing list? I am increasingly worried that with mail services being increasingly centralized across 4-5 major providers we'll soon find it quite difficult to continue using mailing lists and have them reliably deliver mail to all participants. Heck, I've been battling with Gmail over the weekend due to it arbitrarily deciding that Robert Richter is receiving way too much mail.

Git-bug, on the other hand, is a great internal bug tracker for automatically generated bug reports -- it already provides the necessary structured platform, is decentralized, easily clonable and offline-able, and does not suffer from some mail provider arbitrarily deciding throttle its traffic for no obvious reason.

.. [1] https://github.com/MichaelMure/git-bug

Thanks for the pointer. I will play with this and see how this works.

-- Shuah

#12 lore raw
In reply to Konstantin Ryabitsev
Quoted 3 lines · Konstantin Ryabitsev

May I recommend using a project like git-bug [1] instead of a mailing list?

I'm not a huge fan of mailing lists only for bug handling, but on the other hand, I do think that a bug handling thing fundamentally needs

(a) automatic aging out of bug reports

(b) targeted push notifications (ie ractically speaking - email)

and that the email part is important. And it can't just be a "hey, something changed". It needs to contain enough information in the email to give a developer sufficient knowledge that the developer knows whether it's even worth it looking deeper at the bug, and looking at the database (in fact, it should contain enough actual information that for simple issues, the developer should go "D'oh!" and just fix the bug).

We went through this with syzbot already, where the notification was too heavy-handed and not useful. It has improved immensely. There's a back-end with lots more data, but the emails have gone from "illegible data" to "pretty informational", and I think it improved peoples reaction to them a lot.

Apparently syzbot doesn't do well on the (a) side though. Honestly, any system that just keeps things open for 300+ days has something wrong with it. At some point it needs to be automatically either closed, or re-notified. The "just keep it around" is simply not an option.

Linus

#13 lore raw
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Konstantin Ryabitsev

May I recommend using a project like git-bug [1] instead of a mailing list?

I'm not a huge fan of mailing lists only for bug handling, but on the other hand, I do think that a bug handling thing fundamentally needs

(a) automatic aging out of bug reports

(b) targeted push notifications (ie ractically speaking - email)

Quoted 7 lines

and that the email part is important. And it can't just be a "hey, something changed". It needs to contain enough information in the email to give a developer sufficient knowledge that the developer knows whether it's even worth it looking deeper at the bug, and looking at the database (in fact, it should contain enough actual information that for simple issues, the developer should go "D'oh!" and just fix the bug).

Would it make sense to do some bikeshedding around how an ideal email bug report should look like, and then get syzbot and maybe kernelci to switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll have the testing & fuzzing MC going on where this can be discussed and agreed on once there is a concrete proposal.

Quoted 11 lines

We went through this with syzbot already, where the notification was too heavy-handed and not useful. It has improved immensely. There's a back-end with lots more data, but the emails have gone from "illegible data" to "pretty informational", and I think it improved peoples reaction to them a lot.

Apparently syzbot doesn't do well on the (a) side though. Honestly, any system that just keeps things open for 300+ days has something wrong with it. At some point it needs to be automatically either closed, or re-notified. The "just keep it around" is simply not an option.

My understanding is that syzbot closes a bug once the bot sees an annotated commit indicating that it fixes a certain bug. These annotations get lost/forgotten/etc so bugs remain open even after they were fixed.

Maybe we could sync up our bugtrackers with the kernel's release cycle? close all bugs older than a certain kernel version once the corresponding stable kernel goes EoL?

--
Thanks,
Sasha

#14 lore raw
Quoted 10 lines · Sasha Levin
Quoted 5 lines

Apparently syzbot doesn't do well on the (a) side though. Honestly, any system that just keeps things open for 300+ days has something wrong with it. At some point it needs to be automatically either closed, or re-notified. The "just keep it around" is simply not an option.

My understanding is that syzbot closes a bug once the bot sees an annotated commit indicating that it fixes a certain bug. These annotations get lost/forgotten/etc so bugs remain open even after they were fixed.

A lot of such bugs get fixed (directly or totally indirectly) without authors of the fixes being even remotely aware of the syzbot report.

So being able to perform "ok, this is not an issue any more for whatever reason, I am closing myself as RESOLVED" operation would be really nice to have feature, which I think is currently missing.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#15 lore raw
Quoted 4 lines · Jiri Kosina

So being able to perform "ok, this is not an issue any more for whatever reason, I am closing myself as RESOLVED" operation would be really nice to have feature, which I think is currently missing.

Honestly, syzbot - or some related infrastructure - should just do that automatically.

If the syzbot bug is old, and no longer reproduces with modern kernels, then just close it. No human should waste any time on that.

As you say, syzbot can find things that others find independently, so it's not even a "syzbot didn't get the credit, and didn't close things as a result" issue. It can just as well be "bug was fixed without syzbot being directly involved at all, and the syzbot report is just stale old data".

Linus

#16 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 3 lines

So being able to perform "ok, this is not an issue any more for whatever reason, I am closing myself as RESOLVED" operation would be really nice to have feature, which I think is currently missing.

Honestly, syzbot - or some related infrastructure - should just do that automatically.

Quoted 8 lines

If the syzbot bug is old, and no longer reproduces with modern kernels, then just close it. No human should waste any time on that.

As you say, syzbot can find things that others find independently, so it's not even a "syzbot didn't get the credit, and didn't close things as a result" issue. It can just as well be "bug was fixed without syzbot being directly involved at all, and the syzbot report is just stale old data".

Agreed.

At the same time, I definitely don't want this to sound as if syzbot is just reporting random things that people will be fixing anyway. That's definitely not the case in general.

But the volume is simply so overwhelming, that syzbot needs to be able to cope with the fact that bugs are also fixed independently.

That'll help the state / usefulness of the reported bugs a lot IMO.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#17 lore raw
In reply to Sasha Levin
Quoted 7 lines · Sasha Levin

Would it make sense to do some bikeshedding around how an ideal email bug report should look like, and then get syzbot and maybe kernelci to switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll have the testing & fuzzing MC going on where this can be discussed and agreed on once there is a concrete proposal.

I think we'll find that there's a lot of variation in what's useful to report directly in the e-mail and how comes from the particular tests that are being done, what's critical information about the environment will vary quite a bit and there's going to be taste differences as well. A lot of this is a combination of common sense and the reporters being able to be responsive to feedback from their users, there's not much commonality in the formatting of the existing reports from static analysis but they're pretty much all useful (at least the ones I tend to get are useful to me).

There is the question of tooling to track the bugs between multiple bots which would benefit if they were doing standard things but if people are working on that rather than trying to serve both humans and computers at once we might be better off with some sort of API and providing a URL in the emails for the machines to go and look at.

author
#18 lore raw
Quoted 20 lines · Mark Brown
Quoted 7 lines · Sasha Levin

Would it make sense to do some bikeshedding around how an ideal email bug report should look like, and then get syzbot and maybe kernelci to switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll have the testing & fuzzing MC going on where this can be discussed and agreed on once there is a concrete proposal.

I think we'll find that there's a lot of variation in what's useful to report directly in the e-mail and how comes from the particular tests that are being done, what's critical information about the environment will vary quite a bit and there's going to be taste differences as well. A lot of this is a combination of common sense and the reporters being able to be responsive to feedback from their users, there's not much commonality in the formatting of the existing reports from static analysis but they're pretty much all useful (at least the ones I tend to get are useful to me).

Quoted 6 lines

There is the question of tooling to track the bugs between multiple bots which would benefit if they were doing standard things but if people are working on that rather than trying to serve both humans and computers at once we might be better off with some sort of API and providing a URL in the emails for the machines to go and look at.

I am going to bounce a proposal and brace for impact :)

What do people think about managing bugs the way we handle our development workflow?

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

../bugs similar to ../Documentation

- Associate a mailing list - linux-bugs
- Define a format in which bugs should be sent to this list and what should be included in the report.

e.g: [BUG REPORT] sub-system title
Failure mode, release, stack trace (if applicable) etc.

- A group of maintainers/developers (volunteers) take turns to watch this list as a long term plan. Yes, I am proposing a mailing list, consistent with our development model.
- History of the bug including the fix commit ID will be preserved. We will have to figure out what it means to close a bug and general lifetime. There is a big difference between these objects and other source objects. These objects have short lifetime.

I haven't thought through everything yet.

My motivation is that, this way bug reporting and managing will be part of our workflow and becomes part of our process. It helps centralize bug reports.

I will volunteer to start a sandbox repo and take on the role of defining format etc. and maintain it for a release or two, to test the idea. If we like it after the trial, we can adapt it and switch to long term maintainer plan I mentioned above.

thanks,
-- Shuah

#19 lore raw
Quoted 6 lines · Shuah Khan

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

../bugs similar to ../Documentation

If bugs are tracked in the source tree doesn't that make it difficult for people to update the bugs?

#20 lore raw

Hi Mark,

Quoted 10 lines · Mark Brown
Quoted 6 lines · Shuah Khan

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

../bugs similar to ../Documentation

If bugs are tracked in the source tree doesn't that make it difficult for people to update the bugs?

Indeed. While storing the bug state with the sources sounds like a great idea, it's far from trivial in distributed development:
- Where are bugs reported (added to git)?
In upstream, or in maintainer repos?
At least the tree where a fix is applied should carry the bug report.
- Unless downstream trees merge from upstream on a regular basis, they won't have all reports for bugs that apply to their trees.

Closing bugs does follow the expected path, though: both fix and bug report update move upstream through a pull request.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
author
#21 lore raw

Hi Mark and Geert,

Quoted 14 lines · Geert Uytterhoeven

Hi Mark,

Quoted 10 lines · Mark Brown
Quoted 6 lines · Shuah Khan

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

../bugs similar to ../Documentation

If bugs are tracked in the source tree doesn't that make it difficult for people to update the bugs?

Right. It would be harder for users to report bugs. Unless we com up with a way to automate the process of taking user reported generating reports that can be committed.

As I was playing with git-bug, I started to think maybe we can leverage our process to keep bugs in the same repo.

Quoted 8 lines

Indeed. While storing the bug state with the sources sounds like a great idea, it's far from trivial in distributed development:
- Where are bugs reported (added to git)?
In upstream, or in maintainer repos?
At least the tree where a fix is applied should carry the bug report.
- Unless downstream trees merge from upstream on a regular basis, they won't have all reports for bugs that apply to their trees.

My thinking is that we follow the same process of keeping the repos consistent.

Quoted 3 lines

Closing bugs does follow the expected path, though: both fix and bug report update move upstream through a pull request.

This is the value add for me as I proposed the idea. There are lot of logistics to consider as you both pointed out.

thanks,
-- Shuah

#22 lore raw
In reply to Shuah Khan
Quoted 33 lines · Shuah Khan
Quoted 20 lines · Mark Brown
Quoted 7 lines · Sasha Levin

Would it make sense to do some bikeshedding around how an ideal email bug report should look like, and then get syzbot and maybe kernelci to switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll have the testing & fuzzing MC going on where this can be discussed and agreed on once there is a concrete proposal.

I think we'll find that there's a lot of variation in what's useful to report directly in the e-mail and how comes from the particular tests that are being done, what's critical information about the environment will vary quite a bit and there's going to be taste differences as well. A lot of this is a combination of common sense and the reporters being able to be responsive to feedback from their users, there's not much commonality in the formatting of the existing reports from static analysis but they're pretty much all useful (at least the ones I tend to get are useful to me).

Quoted 6 lines

There is the question of tooling to track the bugs between multiple bots which would benefit if they were doing standard things but if people are working on that rather than trying to serve both humans and computers at once we might be better off with some sort of API and providing a URL in the emails for the machines to go and look at.

I am going to bounce a proposal and brace for impact :)

What do people think about managing bugs the way we handle our development workflow?

Quoted 6 lines

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

../bugs similar to ../Documentation

I think that having them in the same tree is going to complicate things more than it will help.

It would probably be simpler to have this in a separate tree (or as git objects inside Linus's tree).

Quoted 6 lines

- Associate a mailing list - linux-bugs
- Define a format in which bugs should be sent to this list and what should be included in the report.

e.g: [BUG REPORT] sub-system title
Failure mode, release, stack trace (if applicable) etc.

Yes! Hopefully something that is both easily machine readable and human readable.

Quoted 3 lines

- A group of maintainers/developers (volunteers) take turns to watch this list as a long term plan. Yes, I am proposing a mailing list, consistent with our development model.

I'd be happy to help here.

Quoted 10 lines

- History of the bug including the fix commit ID will be preserved. We will have to figure out what it means to close a bug and general lifetime. There is a big difference between these objects and other source objects. These objects have short lifetime.

I haven't thought through everything yet.

My motivation is that, this way bug reporting and managing will be part of our workflow and becomes part of our process. It helps centralize bug reports.

Having bugs work the same way like patches is a great approach IMHO. Being able to git-send-email bugs, or to use our various autmation tools for git commits on bugs is really great.

We have a lot of infrastructure written around our existing git/email process, so why not leverage it more as you suggest?

--
Thanks,
Sasha

author
#23 lore raw
Quoted 51 lines · Sasha Levin
Quoted 33 lines · Shuah Khan
Quoted 20 lines · Mark Brown
Quoted 7 lines · Sasha Levin

Would it make sense to do some bikeshedding around how an ideal email bug report should look like, and then get syzbot and maybe kernelci to switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll have the testing & fuzzing MC going on where this can be discussed and agreed on once there is a concrete proposal.

I think we'll find that there's a lot of variation in what's useful to report directly in the e-mail and how comes from the particular tests that are being done, what's critical information about the environment will vary quite a bit and there's going to be taste differences as well. A lot of this is a combination of common sense and the reporters being able to be responsive to feedback from their users, there's not much commonality in the formatting of the existing reports from static analysis but they're pretty much all useful (at least the ones I tend to get are useful to me).

Quoted 6 lines

There is the question of tooling to track the bugs between multiple bots which would benefit if they were doing standard things but if people are working on that rather than trying to serve both humans and computers at once we might be better off with some sort of API and providing a URL in the emails for the machines to go and look at.

I am going to bounce a proposal and brace for impact :)

What do people think about managing bugs the way we handle our development workflow?

Quoted 6 lines

- Create a directory under our source tree for bugs. This way bugs stay with the source and we manage them the same way we handle non-runtime objects: Documenation, scripts etc. This can be flat directory or have a sub-dir structure underneath.

 ../bugs similar to ../Documentation

I think that having them in the same tree is going to complicate things more than it will help.

It would probably be simpler to have this in a separate tree (or as git objects inside Linus's tree).

Quoted 4 lines

- Associate a mailing list - linux-bugs
- Define a format in which bugs should be sent to this list and what should be included in the report.

Quoted 6 lines
Quoted 2 lines

 e.g: [BUG REPORT] sub-system title
 Failure mode, release, stack trace (if applicable) etc.

Yes! Hopefully something that is both easily machine readable and human readable.

Can the AUTO SEL logic be leveraged for updates to bugs?

Quoted 6 lines
Quoted 3 lines

- A group of maintainers/developers (volunteers) take turns to watch this list as a long term plan. Yes, I am proposing a mailing list, consistent with our development model.

I'd be happy to help here.

Awesome.

Quoted 11 lines
Quoted 10 lines

- History of the bug including the fix commit ID will be preserved. We will have to figure out what it means to close a bug and general lifetime. There is a big difference between these objects and other source objects. These objects have short lifetime.

I haven't thought through everything yet.

My motivation is that, this way bug reporting and managing will be part of our workflow and becomes part of our process. It helps centralize bug reports.

Quoted 7 lines

Having bugs work the same way like patches is a great approach IMHO. Being able to git-send-email bugs, or to use our various autmation tools for git commits on bugs is really great.

We have a lot of infrastructure written around our existing git/email process, so why not leverage it more as you suggest?

Right. I am hoping we can leverage a lot of our existing tools and processes if we go with the patch process. Maybe be AUTO SEL logic can also be leveraged and extended to generate bugs reports and/or generate updates for bugs.

thanks,
-- Shuah

#24 lore raw
In reply to Sasha Levin
Quoted 3 lines · Sasha Levin

Maybe we could sync up our bugtrackers with the kernel's release cycle? close all bugs older than a certain kernel version once the corresponding stable kernel goes EoL?

We do something like this for Fedora. After the new kernel gets rebased, we ask the reporters if the bug is still reproducible. This helps prune down some of the one off cases nobody ever saw again as well as issues which were actually fixed. Humans sometimes get annoyed at this as it looks like we're not paying attention but bots wouldn't care if we closed their own issues.

Laura

#25 lore raw
Quoted 11 lines · Laura Abbott
Quoted 3 lines · Sasha Levin

Maybe we could sync up our bugtrackers with the kernel's release cycle? close all bugs older than a certain kernel version once the corresponding stable kernel goes EoL?

We do something like this for Fedora. After the new kernel gets rebased, we ask the reporters if the bug is still reproducible. This helps prune down some of the one off cases nobody ever saw again as well as issues which were actually fixed. Humans sometimes get annoyed at this as it looks like we're not paying attention but bots wouldn't care if we closed their own issues.

Sadly that's reality, we can't fix *all* bugs, so sometimes real bugs would get closed but then hopefully be re-opened again if bots/users are still hitting them.

Maybe they should be closed with a "-ENOREPRO" which would hopefully encourage those users/bots to try and repro the issue on a newer version of the kernel.

--
Thanks,
Sasha

#26 lore raw
In reply to Linus Torvalds
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Konstantin Ryabitsev

May I recommend using a project like git-bug [1] instead of a mailing list?

I'm not a huge fan of mailing lists only for bug handling, but on the other hand, I do think that a bug handling thing fundamentally needs

(a) automatic aging out of bug reports

(b) targeted push notifications (ie ractically speaking - email)

Quoted 7 lines

and that the email part is important. And it can't just be a "hey, something changed". It needs to contain enough information in the email to give a developer sufficient knowledge that the developer knows whether it's even worth it looking deeper at the bug, and looking at the database (in fact, it should contain enough actual information that for simple issues, the developer should go "D'oh!" and just fix the bug).

Yes, I was envisioning a mechanism with two different moving parts -- one the actual bot that would find and create bug entries in a distributed system like git-bug, and another an alert and collaboration tool that would monitor and notify developers when a bug is created matching the parameters they have defined. For example, Greg can define a query matching "all bugs mentioning usb" and attach the action of "send bug summary to the mailing list." We can run the matching and notification daemon at kernel.org, or anyone can set it up to run it on their own system to fire as a post-update hook on when they pull the git-bug repo.

These are the upsides for a distributed system like this:

1. Doesn't require a stable internet connection when running on the laptop of a developer who's out in a remote corner of the world (except for the actual pull to fetch new data).
2. Provides full bug history for the project that doesn't depend on kernel.org not falling off the face of the earth.
3. Offers ability to easily analyze the data, including historical (the same way public-inbox on lore.kernel.org/lkml allows anyone to crunch full kernel development history).
4. Can hook into other bugtracking systems like bugzilla, Gitlab, or Github via one-way or two-way bridges. E.g. instead of "send an email to the list" for USB bugs, the action can be "create a bug on bugzilla.kernel.org and sync back comments and bug status changes." The bridges for launchpad and GitHub already exist for git-bug.
5. Bots can check for duplicates before creating a new bug (I'm not 100% sure how well this would work, but theoretically this is possible). This includes bugs created by other bots, as long as they all feed the same distributed tracker.
6. Bots don't have to develop notification tooling and can concentrate on just creating and pushing new bugs. This lowers the entry barrier if new bots are developed.

It's entirely possible that git-bug is the wrong tool for this -- and, in fact, it's easy to argue that git is the wrong tool for this and what git-bug is doing to twist git to suit its purposes is wrong and horrible. Some would argue that blockchains were envisioned for exactly this purpose, except that blockchain as a technology is currently in the "sardonically cringe whenever someone mentions it" stage of adoption (especially every time bitcoin drops a few thousand points).

In general, I believe that we should be aiming to develop tools that are git-like in the sense that they do not depend on a centralized entity. For the longest time, the distributed nature of email was what made it possible for Linux development to remain truly decentralized, but email in 2019 is radically different from email in 2009. Running an independent email server that reliably sends and receives email is becoming more and more difficult now that most of email traffic goes through 5-6 major companies -- you must do SPF, DKIM, ARC/DMARC, TLS, and who knows what next, just to be reliably accepted by Gmail (maybe). And even if someone uses one of those major email providers doesn't mean the patches won't end up in someone's spam folder, show up a week late, or arrive mangled.

Public-inbox makes this situation a bit better, but it still relies on email remaining generally functional. My hope it that we'll be able to take this a step further and create truly distributed alternatives to platforms like Github/Gitlab that wouldn't require using centralized resources for bug tracking and CI. We've done this for source code with git, and we should aim to do the same for the rest.

-K

author
#27 lore raw
Quoted 7 lines · Konstantin Ryabitsev
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Konstantin Ryabitsev

May I recommend using a project like git-bug [1] instead of a mailing list?

I experimented with git-bug a bit. Doesn't look like it is mature enough for our use. It depends on golang-go

I would like to have the option to build from sources, the instructions are rather poor and couldn't build following those.

Is there another git-bug like option we can look into?

thanks,
-- Shuah

#28 lore raw
Quoted 11 lines · Shuah Khan
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

May I recommend using a project like git-bug [1] instead of a mailing list?

I experimented with git-bug a bit. Doesn't look like it is mature enough for our use. It depends on golang-go

I would like to have the option to build from sources, the instructions are rather poor and couldn't build following those.

Is there another git-bug like option we can look into?

If there exists one, I'm unaware of it, unfortunately. The right person to talk to about developing something like this would be the author of Public-Inbox (Eric Wong), who digs distributed systems and would probably love working on something like this.

However, I have been unsuccessful in getting LF to fund such development projects, and he's certainly not in a position to work on something like this for free.

Hey, you work at LF now. Maybe you'll have more luck convincing The Right People that this is important work that would greatly benefit the Linux community? :)

-K

author
#29 lore raw
Quoted 11 lines · Konstantin Ryabitsev
Quoted 10 lines · Shuah Khan
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

May I recommend using a project like git-bug [1] instead of a mailing list?

I experimented with git-bug a bit. Doesn't look like it is mature enough for our use. It depends on golang-go

I would like to have the option to build from sources, the instructions are rather poor and couldn't build following those.

I was able to compile with "go" under my home directory finally and run a few commands on git-bug repo. There is the next step of trying to see if it works for other repos.

I still have my concerns about dependency on "go" and I would like to get this to work under a non-home directory structure.

Quoted 4 lines
Quoted 1 line

Is there another git-bug like option we can look into?

If there exists one, I'm unaware of it, unfortunately. The right person to talk to about developing something like this would be the author of Public-Inbox (Eric Wong), who digs distributed systems and would probably love working on something like this.

Okay.

Quoted 4 lines

However, I have been unsuccessful in getting LF to fund such development projects, and he's certainly not in a position to work on something like this for free.

Hey, you work at LF now. Maybe you'll have more luck convincing The Right People that this is important work that would greatly benefit the Linux community? :)

:)

My thoughts on this are: "we needed a solution yesterday" and we can't start building a shelter for rain while its raining. We need some interim solution, even if we choose to explore a long term project development idea.

thanks,
-- Shuah

#30 lore raw
In reply to Shuah Khan
Quoted 46 lines · Shuah Khan
Quoted 39 lines · Sasha Levin
Quoted 30 lines · Laura Abbott
Quoted 21 lines · Shuah Khan

I would like to propose a topic to discuss ideas to keep up with Syzbot bugs on an ongoing basis. Good news is, as of this writing, we have 1276 fixed, 86 in moderation, and 62 fix pending. However, there are 523 open bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other developers. In addition, I included syzbot bug analysis as a required contribution in the Linux Kernel Mentorship Program application process. This is for learning debugging skills as well as getting help towards fixing bugs. It has been successful in getting a few fixes in. My goal is  to make these efforts a bit more structured to focus on areas that need help.

There are some challenges, the obvious ones being finding time to analyze, reproduce, and fix. Reproducers are available in many cases, and these bugs can be reproduced. Identifying the right fix and fixing is the overwhelming part with the number of outstanding problems.

Quoted 7 lines

My objective for this topic is to explore and identify areas that might need help in analyzing and fixing bugs in general and especially the ones that have been open for a while. It would help me channel and focus the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Quoted 7 lines

I'm interested in this topic for not just syzbot but other bug trackers as well. Fedora gets a steady flow of bugs filed and the three official maintainers do their best to review but sometimes things slip through, especially if we hit a time when several of us are out or traveling. The kernel.org bugzilla also gets a number of bug reports that sometimes get lost. I'd love to see if there's a process that could work for syzbot and other high volume kernel trackers.

Quoted 5 lines

Maybe the solution here is to standardize bug reporting bots? A [email protected] mailing list which our various bots could report to with a standard format that'll allow users/maintainers to easily filter it to get only bugs they care about?

Quoted 4 lines

Easy to filter and manage is important for usability. I mentioned in my response Kees that I am finding it hard to update syzbot bug status to mark them as duplicate and/or fixed. The email interface is nice for individual bugs, but not very easy when it comes to other actions.

Sure, there are quite a few drawbacks to the mailing-list-as-bugtracker approach, but I feel that we need to start with something and either build on top of that or replace it in the future with something better.

I don't think that there's a bugtracker in existence today that would be perfect for our "weird" usecase.

--
Thanks,
Sasha