Skip to contentSkip to conversation

Regression tracking

Ksummit-discuss Takashi Iwai 8 participants 15 messages Jul 26, 2016 – Aug 10, 2016

lore
Share & tools
lore
#0 lore raw

As this topic doesn't seem posted yet, let me pitch.

Until recently, we've had no regression tracking since Rafael stepped down from the maintenance of regression list. Now fortunately, Thorsten started gathering and posting the regression reports on ML again, and this helps already a lot.

Though, it immediately raised me a few things to be discussed:

- How can we keep it rolling stably at this time?
- How can each developer / maintainer help or be involved better?
- Can we provide better services than only a plain text list on ML?

Also, since the regression on stable trees was a hot topic:

- Should the regression tracking cover the stable trees? How well does it scale?

thanks,

Takashi

#1 lore raw
Quoted 1 line · Takashi Iwai

As this topic doesn't seem posted yet, let me pitch.

I'm interested in this one.

Quoted 2 lines

Until recently, we've had no regression tracking since Rafael stepped down from the maintenance of regression list.

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Quoted 8 lines

Now fortunately, Thorsten started gathering and posting the regression reports on ML again, and this helps already a lot.

Though, it immediately raised me a few things to be discussed:

- How can we keep it rolling stably at this time?
- How can each developer / maintainer help or be involved better?
- Can we provide better services than only a plain text list on ML?

I think it is possible, but it also depends on how much time the person who tracks regressions can spend on it.

It may be automated somewhat (from experience), but still involves quite a bit of manual work.

The fact that the list is only sent by email makes it a bit hard to follow (you always need to look for that particular email in your mailbox) and if you are a maintainer, it isn't easy to say upfront which bugs may be relevant to you without checking them one by one.

That's why I was sending bug-specific messages to maintainers too when I was tracking regressions (it was possible ot opt-out from that, but I'm not sure whether or not that was a good idea).

Quoted 6 lines

Also, since the regression on stable trees was a hot topic:

- Should the regression tracking cover the stable trees? How well does it scale?

From experience, work needed grows linearly with the number of releases you

are trying to track at the same time. I've never tracked more than two or three (maybe) at a time IIRC, because of exactly that.

Thanks,
Rafael

#2 lore raw

(@Takashi & @Rafel: repost after subscribing to ksummit-discuss, which is moderated for non-members :-/ )

Quoted 8 lines · Rafael J. Wysocki
Quoted 8 lines · Takashi Iwai
Quoted 1 line
Quoted 1 line

As this topic doesn't seem posted yet, let me pitch.

I'm interested in this one.

Quoted 2 lines
Quoted 2 lines

Until recently, we've had no regression tracking since Rafael stepped down from the maintenance of regression list.

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what I can see it's a lot of manual and boring work that only now and then feels rewarding :-/

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Now fortunately, Thorsten started gathering and posting the regression reports on ML again, and this helps already a lot.

Though, it immediately raised me a few things to be discussed:

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

- How can we keep it rolling stably at this time?
- How can each developer / maintainer help or be involved better?
- Can we provide better services than only a plain text list on ML?

I think it is possible, but it also depends on how much time the person who tracks regressions can spend on it.

It may be automated somewhat (from experience), but still involves quite a bit of manual work.

My rough plan after doing regression tracking for a few weeks now: Look a bit closer at patchwork and check if it can be adjusted to automate the boring parts of regression tracking. I have a mostly finished mail that has some more details on what got me thinking into that direction, but I'm on holiday till Friday and wanted to send it out once I'm home again.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Also, since the regression on stable trees was a hot topic:
- Should the regression tracking cover the stable trees? How well does it scale?

From experience, work needed grows linearly with the number of releases you are trying to track at the same time. I've never tracked more than two or three (maybe) at a time IIRC, because of exactly that.

Yeah, without more manpower or some automation (like bitkeeper/git made Linus scale better) it might lead to a quick burn out if one wants to track more then two releases.

But OTOH I think it would be really good to track at least the latest stable tree, as quite a few people only start testing kernel versions like 4.7 once they got released. Sure, it's not as easy to revert changes after that point, but it worthwhile nevertheless afaics.

Ciao, Thorsten

#3 lore raw
Quoted 8 lines · Thorsten Leemhuis
Quoted 1 line · Rafael J. Wysocki
Quoted 1 line · Takashi Iwai

My rough plan after doing regression tracking for a few weeks now: Look a bit closer at patchwork and check if it can be adjusted to automate the boring parts of regression tracking. I have a mostly finished mail that has some more details on what got me thinking into that direction, but I'm on holiday till Friday and wanted to send it out once I'm home again.

FWIW, I posted my mail to LKML now:
http://lkml.kernel.org/r/[email protected]

Some of the thoughts and ideas mentioned there might be relevant for a kernel summit discussion about regression tracking; let me know if there is anything else I could do to help.

Ciao, Thorsten

#4 lore raw
In reply to Thorsten Leemhuis
Quoted 8 lines · Thorsten Leemhuis
Quoted 2 lines · Rafael J. Wysocki
Quoted 2 lines

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what I can see it's a lot of manual and boring work that only now and then feels rewarding :-/

I am also interested in this one. Regression tracking is a very time consuming task and we should discuss how that can be improved, so that we get a sustainable solution that works for the kernel.

As Thorsten already pointed out, better tooling would help. But I think this will also involve teaching our testers/users on how to best report regressions. Part of the discussion should also be on automation, for example if we could/should revive the kerneloops tool?

Just my few thoughts, but if this is goind to be discussed, I'd like to particpate.

Joerg

#5 lore raw
Quoted 11 lines · Joerg Roedel
Quoted 7 lines · Thorsten Leemhuis
Quoted 2 lines · Rafael J. Wysocki
Quoted 2 lines

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what I can see it's a lot of manual and boring work that only now and then feels rewarding :-/

[…] But I think
this will also involve teaching our testers/users on how to best report regressions.

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

Quoted 2 lines

Part of the discussion should also be on automation, for example if we could/should revive the kerneloops tool?

Ugh, what's the latest status of it? I just saw
http://www.kerneloops.org/ is kind of alive again as
http://oops.kernel.org/ (I had missed that) and has data from recent kernel versions. But it seems the code behind it hasn't seen much changes in the last two years:
https://github.com/oops-kernel-org/kerneloops/commits/master

Side note: kernelopps and error reporting was talked about on KS2013:
https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these days by default installs abrt-addon-kerneloops (see
https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
for a very brief intro). Things it picks up can be seen on
https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package maintainers use the data afaik, but I don't know any more details.
Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not something other distros can easily integrate like it happened with the original kerneloops tool years ago.

Quoted 1 line

[...]

Ciao, Thorsten

#6 lore raw
Quoted 14 lines · Thorsten Leemhuis
Quoted 11 lines · Joerg Roedel
Quoted 7 lines · Thorsten Leemhuis
Quoted 2 lines · Rafael J. Wysocki
Quoted 2 lines

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what I can see it's a lot of manual and boring work that only now and then feels rewarding :-/

[…] But I think
this will also involve teaching our testers/users on how to best report regressions.

Quoted 7 lines

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

Right, unfortunately Bugzilla is often left uncared although we've already got a regression report.

Meanwhile there is a regression flag already in bugzilla.kernel.org, and this could be automatically sending a notification to you. Then you can ping the corresponding maintainer.

Quoted 10 lines
Quoted 2 lines

Part of the discussion should also be on automation, for example if we could/should revive the kerneloops tool?

Ugh, what's the latest status of it? I just saw
http://www.kerneloops.org/ is kind of alive again as
http://oops.kernel.org/ (I had missed that) and has data from recent kernel versions. But it seems the code behind it hasn't seen much changes in the last two years:
https://github.com/oops-kernel-org/kerneloops/commits/master

Quoted 10 lines

Side note: kernelopps and error reporting was talked about on KS2013:
https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these days by default installs abrt-addon-kerneloops (see
https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
for a very brief intro). Things it picks up can be seen on
https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package maintainers use the data afaik, but I don't know any more details.
Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not something other distros can easily integrate like it happened with the original kerneloops tool years ago.

Yeah, I'm afraid that it's too heavy-weight. And overall, I guess you'd get way too many noises by that, more than the really useful data.

IMO, the possible tactics is to educate subsystem maintainers to properly report regressions. Ideally, you wouldn't have to handle any bugs by yourself, but just let your slaves debugging, and report back to you the status :)

Takashi

#7 lore raw
In reply to Thorsten Leemhuis
Quoted 7 lines · Thorsten Leemhuis

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

In my case, I am only looking into the kernel.org bugzilla if someone pings me about it. Usually I get the upstream bug reports via email. And this is part of the problem, that we have multiple ways that we teach users how to report bugs/regressions. That really increases the effort needed for you to compile the regression list and keep track of everything in there.

So the right step here is to discuss how we want to get bugs reported and settle on a single way we tell the users and testers. From there it is much easier to get the reports dispatched to the right people.

Quoted 6 lines

Ugh, what's the latest status of it? I just saw
http://www.kerneloops.org/ is kind of alive again as
http://oops.kernel.org/ (I had missed that) and has data from recent kernel versions. But it seems the code behind it hasn't seen much changes in the last two years:
https://github.com/oops-kernel-org/kerneloops/commits/master

Interesting, I wasn't aware of that either. From a first look it seems that the reporting tool is probably not widely used, at least not like the Fedora tool.

Quoted 3 lines

Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not something other distros can easily integrate like it happened with the original kerneloops tool years ago.

Well, but maybe parts of that software could be re-used for our regression tracking. The oops-parser is really cool, as it extracts the stack-traces and finds the source-files the symbols belong to.

You can extract the same information from LKML bug-reports and use the file-names together with the MAINTAINERS file to automatically compile an initial list of potential developers who might look at the report.

To further automate this, it could be put into a mail-filter which is called by a procmail rule against LKML mails and that spits out potential bug/regression reports.

Certainly a few heuristics are needed too to improve the signal-to-noise ratio, but over time this might become a helpful automation step.

Joerg

#8 lore raw
Quoted 16 lines · Joerg Roedel
Quoted 7 lines · Thorsten Leemhuis

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

In my case, I am only looking into the kernel.org bugzilla if someone pings me about it. Usually I get the upstream bug reports via email. And this is part of the problem, that we have multiple ways that we teach users how to report bugs/regressions. That really increases the effort needed for you to compile the regression list and keep track of everything in there.

Quoted 3 lines

So the right step here is to discuss how we want to get bugs reported and settle on a single way we tell the users and testers. From there it is much easier to get the reports dispatched to the right people.

I need to mention here that PM and ACPI regressions in bugzilla.kernel.org are actually tracked by a team at Intel. We track all of the PM and ACPI bugs filed via the BZ, including regressions, and we're going to be doing that.

Thanks,
Rafael

#9 lore raw
In reply to Joerg Roedel
Quoted 11 lines · Joerg Roedel
Quoted 7 lines · Thorsten Leemhuis

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

In my case, I am only looking into the kernel.org bugzilla if someone pings me about it. Usually I get the upstream bug reports via email.

You can have bugzilla.kernel.org configured to send your list email when someone reports a new bug.

It used to actually be useful: you could reply and it would capture that in the bugzilla entry. However, since the migration to bugzilla version 3, this has started to fail, so you need to add the reporter's cc directly as well now.

Once you get this configuration set up it means that bugzilla and email can be directly equivalent, it's just that only email will have the complete bug text (because bugzilla is lossy on the replies).

James

#10 lore raw
Quoted 11 lines · James Bottomley

You can have bugzilla.kernel.org configured to send your list email when someone reports a new bug.

It used to actually be useful: you could reply and it would capture that in the bugzilla entry. However, since the migration to bugzilla version 3, this has started to fail, so you need to add the reporter's cc directly as well now.

Once you get this configuration set up it means that bugzilla and email can be directly equivalent, it's just that only email will have the complete bug text (because bugzilla is lossy on the replies).

That would be start, at least. For my case, there is not even an IOMMU category yet, I'll try to open a bug to change that and see what happens :)

Joerg

#11 lore raw
In reply to Joerg Roedel
Quoted 6 lines · Joerg Roedel

In my case, I am only looking into the kernel.org bugzilla if someone pings me about it. Usually I get the upstream bug reports via email. And this is part of the problem, that we have multiple ways that we teach users how to report bugs/regressions. That really increases the effort needed for you to compile the regression list and keep track of everything in there.

How about adding an entry, say "B:", to MAINTAINERS to let each subsystem/driver specify the preferred method of reporting bugs? For example, most drivers/gpu/drm bugs would get more attention at
https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#12 lore raw
Quoted 4 lines · Jani Nikula

How about adding an entry, say "B:", to MAINTAINERS to let each subsystem/driver specify the preferred method of reporting bugs? For example, most drivers/gpu/drm bugs would get more attention at
https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.

I thought about that too, it might make sense if we can't agree on anything better. In the end it is more hassle for the users and requires more knowledge on their side. They have to know at least in which part of the kernel the problem occured and then look that part up in the MAINTAINERS file to find out how to report the bug. There are probably quite a few users who give up somewhere on that path.

Joerg

#13 lore raw
Quoted 6 lines · Joerg Roedel
Quoted 4 lines · Jani Nikula

How about adding an entry, say "B:", to MAINTAINERS to let each subsystem/driver specify the preferred method of reporting bugs? For example, most drivers/gpu/drm bugs would get more attention at
https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.

Quoted 6 lines

I thought about that too, it might make sense if we can't agree on anything better. In the end it is more hassle for the users and requires more knowledge on their side. They have to know at least in which part of the kernel the problem occured and then look that part up in the MAINTAINERS file to find out how to report the bug. There are probably quite a few users who give up somewhere on that path.

We already have this trouble on the gfx side, just worse: Users generally have no idea which one of kernel driver, X driver, mesa/gl, libva/video or whatever exactly blew up. Which is why we're redirecting everyone to bugs.freedesktop.org, there we can at elast reassign between the different parts.

Of course that then means we can't easily reassing to other kernel components, but that seems to happen far less often (for us).

Anyway, +1 from me for B: in MAINTAINERS to make it at least possible for users to figure out where to file a bug. Default would be mailo:
<mailing list from same maintainer entry>.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#14 lore raw
In reply to Thorsten Leemhuis
Quoted 31 lines · Thorsten Leemhuis
Quoted 11 lines · Joerg Roedel
Quoted 7 lines · Thorsten Leemhuis
Quoted 2 lines · Rafael J. Wysocki
Quoted 2 lines

There were other people trying to do that when I had stopped doing it, but they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what I can see it's a lot of manual and boring work that only now and then feels rewarding :-/

[…] But I think
this will also involve teaching our testers/users on how to best report regressions.

Yeah, definitely. One afaics important part that needs to be done in this area: make bugzilla.kernel.org work better. At least some (or maybe even quite a lot?) of the bugs reported there afaics seem to not get proper attention from the right people. Due to that I'd assume some testers will have come to conclusions like "testing for regressions and bugs is not worth the trouble, as the kernel developers are not interested in the bugs I file anyway".

Quoted 2 lines

Part of the discussion should also be on automation, for example if we could/should revive the kerneloops tool?

Ugh, what's the latest status of it? I just saw
http://www.kerneloops.org/ is kind of alive again as
http://oops.kernel.org/ (I had missed that) and has data from recent kernel versions. But it seems the code behind it hasn't seen much changes in the last two years:
https://github.com/oops-kernel-org/kerneloops/commits/master

Quoted 10 lines

Side note: kernelopps and error reporting was talked about on KS2013:
https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these days by default installs abrt-addon-kerneloops (see
https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
for a very brief intro). Things it picks up can be seen on
https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package maintainers use the data afaik, but I don't know any more details.
Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not something other distros can easily integrate like it happened with the original kerneloops tool years ago.

Fedora uses ABRT but the data tends to be fairly noisy for kernel regression tracking from my experience. It gets flooded with reports from old kernel versions and issues that have been there for a while. It's certainly possible to use the data and I'm glad it's collected but it still requires more manual thinking and analysis than I would like. What's much more useful with ABRT is the one click "file a bugzilla" and duplication detection.

Thanks,
Laura