docs: add two texts covering regressions
Thorsten Leemhuis
maintainerv5 cover docs: add two texts covering regressions
"We don't cause regressions" might be the first rule of Linux kernel development, but it and other aspects of regressions nevertheless are hardly described in the Linux kernel's documentation. The following patches change this by creating two documents dedicated to the topic.
The second patch could easily be folded into the first one, but was kept separate, as it might be a bit controversial. This also allows the patch description to explain some backgrounds for this part of the document. Additionally, ACKs and Reviewed-by tags can be collected separately this way.
v5 (this version):
- rename the two documents after review feedback from Jon
- drop the reviewed-by and ACK in the second patch: I slightly changed one entry (the one (the one starting with 'Aim to fix regressions within one week') and added one (the one starting with 'Developers should handle regressions in all supported kernel series') after review feedback from Jon
- remind developers to use "Fixes:" and "Cc: [email protected]" tags in "What's important when fixing regressions"
- drop a few quotes from Linus
- while modifying Documentation/admin-guide/reporting-issues.rst remove quotes around doc references
- a few small adjustments and fixes along the way
v4 (https://lore.kernel.org/lkml/[email protected]/):
- countless small and medium changes after review feedback from Jon (thx), which also lead to a big change:
- split the document into two, one for users and one for developers (both added by the first patch, as they are interlinked).
- fixed and improved a bunch of areas I stumbled upon while checking the text again after the split
- add a third patch to get one of the user-centric document on regressions mentioned in Documentation/admin-guide/reporting-issues.rst
- note: the content added by the second patch did not change significantly, that's why I left an earlier reviewed-by for the patch and an ACK for the series in place there, but dropped the ACK for the first patch of the series
v3 (https://lore.kernel.org/regressions/[email protected]/):
- drop RFC tag
- heavily reshuffled and slightly adjusted the text in the sections "The important bits for people fixing regressions" and "How to add a regression to regzbot's tracking somebody else reported?" to make them easier to grasp
- a few small fixes and improvements
- add ACK for the series from Greg (now for real)
v2/RFC (https://lore.kernel.org/linux-doc/[email protected]/):
- a lot of small fixes, most are for spelling mistakes and grammar errors/problems pointed out in the review feedback I got so far
- add ACK for the series from Greg
v1/RFC (https://lore.kernel.org/linux-doc/[email protected]/):
- initial version
---
Hi! Here is the latest version of my patch-set adding documentation regarding regressions. It's mostly fine-tuning after the latest review feedback from Jon.
@Greg, @Linus, would be cool if you could at least take a look at the second patch and ACK or comment on it to make sure the text is really okay for you, now that Jon indicated he's willing to apply this, unless objections show up.
Ciao, Thorsten
Thorsten Leemhuis (3):
docs: add two documents about regression handling
docs: *-regressions.rst: explain how quickly issues should be handled
docs: reporting-issues.rst: link new document about regressions
Documentation/admin-guide/index.rst | 1 + .../admin-guide/reporting-issues.rst | 73 +- .../admin-guide/reporting-regressions.rst | 451 +++++++++++ .../process/handling-regressions.rst | 746 ++++++++++++++++++ Documentation/process/index.rst | 1 + MAINTAINERS | 2 + 6 files changed, 1237 insertions(+), 37 deletions(-) create mode 100644 Documentation/admin-guide/reporting-regressions.rst create mode 100644 Documentation/process/handling-regressions.rst
base-commit: f647de4b02dcb1815fb3019f86a001a681daf0a1
Signature · 1 line
2.35.1
Thorsten Leemhuis
maintainerPATCH v5 1/3 docs: add two documents about regression handling
Create two documents explaining various aspects around regression handling and tracking; one is aimed at users, the other targets developers.
The texts among others describes the first rule of Linux kernel development and what it means in practice. They also explain what a regression actually is and how to report one properly.
Both texts additionally provide a brief introduction to the bot the kernel's regression tracker uses to facilitate the work, but mention the use is optional.
To sum things up, provide a few quotes from Linus in the document for developers to show how serious we take regressions.
--- Documentation/admin-guide/index.rst | 1 + .../admin-guide/reporting-regressions.rst | 439 ++++++++++++ .../process/handling-regressions.rst | 659 ++++++++++++++++++ Documentation/process/index.rst | 1 + MAINTAINERS | 2 + 5 files changed, 1102 insertions(+) create mode 100644 Documentation/admin-guide/reporting-regressions.rst create mode 100644 Documentation/process/handling-regressions.rst
Documentation/admin-guide/index.rst +1 -0
@@ -35,6 +35,7 @@ problems and bugs in particular. :maxdepth: 1 reporting-issues + reporting-regressions security-bugs bug-hunting bug-bisect
Documentation/admin-guide/reporting-regressions.rst new file +439 -0
439 lines not on this page · raw message
Documentation/process/handling-regressions.rst new file +659 -0
659 lines not on this page · raw message
Documentation/process/index.rst +1 -0
@@ -25,6 +25,7 @@ Below are the essential guides that every developer should read. code-of-conduct-interpretation development-process submitting-patches + handling-regressions programming-language coding-style maintainer-handbooks
MAINTAINERS +2 -0
@@ -10438,6 +10438,8 @@ KERNEL REGRESSIONS M: Thorsten Leemhuis <[email protected]> L: [email protected] S: Supported +F: Documentation/admin-guide/reporting-regressions.rst +F: Documentation/process/handling-regressions.rst KERNEL SELFTEST FRAMEWORK M: Shuah Khan <[email protected]>
Signature · 1 line
2.35.1
Thorsten Leemhuis
maintainerPATCH v5 2/3 docs: *-regressions.rst: explain how quickly issues should be handled
In reply to Thorsten LeemhuisAdd a section with a few rules of thumb about how quickly developers should address regressions to Documentation/process/handling-regressions.rst; additionally, add a short paragraph about this to the companion document Documentation/admin-guide/reporting-regressions.rst as well.
The rules of thumb were written after studying the quotes from Linus found in handling-regressions.rst and especially influenced by statements like "Users are literally the _only_ thing that matters" and "without users, your program is not a program, it's a pointless piece of code that you might as well throw away". The author interpreted those in perspective to how the various Linux kernel series are maintained currently and what those practices might mean for users running into a regression on a small or big kernel update.
That for example lead to the paragraph starting with "Aim to get fixes for regressions mainlined within one week after identifying the culprit, if the regression was introduced in a stable/longterm release or the devel cycle for the latest mainline release". Some might see this as pretty high bar, but on the other hand something like that is needed to not leave users out in the cold for too long -- which can quickly happen when updating to the latest stable series, as the previous one is normally stamped "End of Life" about three or four weeks after a new mainline release. This makes a lot of users switch during this timeframe. Any of them thus risk running into regressions not promptly fixed; even worse, once the previous stable series is EOLed for real, users that face a regression might be left with only three options:
(1) continue running an outdated and thus potentially insecure kernel version from an abandoned stable series
(2) run the kernel with the regression
(3) downgrade to an earlier longterm series still supported
This is better avoided, as (1) puts users and their data in danger, (2) will only be possible if it's a minor regression that doesn't interfere with booting or serious usage, and (3) might be regression itself or impossible on the particular machine, as the users might require drivers or features only introduced after the latest longterm series branched of.
In the end this lead to the aforementioned "Aim to fix regression within one week" part. It's also the reason for the "Try to resolve any regressions introduced in the current development cycle before its end.".
---
Hi! A lot of developers are doing a good job in fixing regressions in a reasonable time span, but I noticed it sometimes takes many weeks to get even simple fixes for regressions merged. Most of the time this is due to one of these factors:
* it takes a long time to get the fix ready, as some developers apparently don't prioritize work on fixing regressions
* fully developed fixes linger in git trees of maintainers for weeks, sometimes even without the fix being in linux-next
This afaics is especially a problem for regressions introduced in mainline, but only found after a new versions was released and a new stable kernel series derived from it. Sometimes fixes for these regressions are even left lying around for weeks until the next merge window, which contributes to a huge pile of fixes getting backported to stable and longterm releases after a merge window ended. Asking developers to speed things up rarely helped, as people have different opinions on how fast regression fixes need to be developed and merged upstream.
That's why it would be a great help to my work as regression tracker if we had some rough written down guidelines for handling regressions, as proposed by the patch below. I'm well aware that the text sets a pretty high bar. That's because I approached the problem primarily from the point of a user, as can be seen by the patch description.
The text added by this patch likely will lead to some discussions, that's why I submit it separately from the rest of the new documents on regressions, which are found in patch 1/3; I also CCed Linus on this patch and hope he states his opinion or even ACKs is. In the end I can easily tone this down or write something totally different: that's totally fine for me, I'm mainly interested in having some expectations roughly documented to get everyone on the same page.
Ciao, Thorsten
--- .../admin-guide/reporting-regressions.rst | 12 +++ .../process/handling-regressions.rst | 87 +++++++++++++++++++ 2 files changed, 99 insertions(+)
Documentation/admin-guide/reporting-regressions.rst +12 -0
@@ -214,6 +214,18 @@ your report on the radar of these people by CCing or forwarding each report to the regressions mailing list, ideally with a "regzbot command" in your mail to get it tracked immediately. +How quickly are regressions normally fixed? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Developers should fix any reported regression as quickly as possible, to provide +affected users with a solution in a timely manner and prevent more users from +running into the issue; nevertheless developers need to take enough time and +care to ensure regression fixes do not cause additional damage. + +The answer thus depends on various factors like the impact of a regression, its +age, or the Linux series in which it occurs. In the end though, most regressions +should be fixed within two weeks. + Is it a regression, if the issue can be avoided by updating some software? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Documentation/process/handling-regressions.rst +87 -0
99 lines not on this page · raw message
Signature · 1 line
2.35.1
Thorsten Leemhuis
maintainerPATCH v5 3/3 docs: reporting-issues.rst: link new document about regressions
In reply to Thorsten LeemhuisMake Documentation/admin-guide/reporting-issues.rst point to the newly created document about regressions (Documentation/admin-guide/regressions-regressions.rst). This allows to shorten a few explanations the new document describes better and in more detail.
While at it move the copyright hint to the end of the file and remove quotes around links to other places in the documentation. Both issues came up during the review of the new documents about regressions.
--- .../admin-guide/reporting-issues.rst | 73 +++++++++---------- 1 file changed, 36 insertions(+), 37 deletions(-)
Documentation/admin-guide/reporting-issues.rst +36 -37
@@ -1,14 +1,5 @@ .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) -.. - If you want to distribute this text under CC-BY-4.0 only, please use 'The - Linux kernel developers' for author attribution and link this as source: - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst -.. - Note: Only the content of this RST file as found in the Linux kernel sources - is available under CC-BY-4.0, as versions of this text that were processed - (for example by the kernel's build system) might contain content taken from - files which use a more restrictive license. - +.. See the bottom of this file for additional redistribution information. Reporting issues ++++++++++++++++
@@ -395,22 +386,16 @@ fixed as soon as possible, hence there are 'issues of high priority' that get handled slightly differently in the reporting process. Three type of cases qualify: regressions, security issues, and really severe problems. -You deal with a 'regression' if something that worked with an older version of -the Linux kernel does not work with a newer one or somehow works worse with it. -It thus is a regression when a WiFi driver that did a fine job with Linux 5.7 -somehow misbehaves with 5.8 or doesn't work at all. It's also a regression if -an application shows erratic behavior with a newer kernel, which might happen -due to incompatible changes in the interface between the kernel and the -userland (like procfs and sysfs). Significantly reduced performance or -increased power consumption also qualify as regression. But keep in mind: the -new kernel needs to be built with a configuration that is similar to the one -from the old kernel (see below how to achieve that). That's because the kernel -developers sometimes can not avoid incompatibilities when implementing new -features; but to avoid regressions such features have to be enabled explicitly -during build time configuration. +You deal with a regression if some application or practical use case running +fine with one Linux kernel works worse or not at all with a newer version +compiled using a similar configuration. The document +Documentation/admin-guide/reporting-regressions.rst explains this in more +detail. It also provides a good deal of other information about regressions you +might want to be aware of; it for example explains how to add your issue to the +list of tracked regressions, to ensure it won't fall through the cracks. What qualifies as security issue is left to your judgment. Consider reading -'Documentation/admin-guide/security-bugs.rst' before proceeding, as it +Documentation/admin-guide/security-bugs.rst before proceeding, as it provides additional details how to best handle security issues. An issue is a 'really severe problem' when something totally unacceptably bad
@@ -517,7 +502,7 @@ line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was not tainted when it noticed the problem; it was tainted if you see 'Tainted:' followed by a few spaces and some letters. -If your kernel is tainted, study 'Documentation/admin-guide/tainted-kernels.rst' +If your kernel is tainted, study Documentation/admin-guide/tainted-kernels.rst to find out why. Try to eliminate the reason. Often it's caused by one these three things:
@@ -1043,7 +1028,7 @@ down the culprit, as maintainers often won't have the time or setup at hand to reproduce it themselves. To find the change there is a process called 'bisection' which the document -'Documentation/admin-guide/bug-bisect.rst' describes in detail. That process +Documentation/admin-guide/bug-bisect.rst describes in detail. That process will often require you to build about ten to twenty kernel images, trying to reproduce the issue with each of them before building the next. Yes, that takes some time, but don't worry, it works a lot quicker than most people assume.
@@ -1073,10 +1058,11 @@ When dealing with regressions make sure the issue you face is really caused by the kernel and not by something else, as outlined above already. In the whole process keep in mind: an issue only qualifies as regression if the -older and the newer kernel got built with a similar configuration. The best way -to archive this: copy the configuration file (``.config``) from the old working -kernel freshly to each newer kernel version you try. Afterwards run ``make -olddefconfig`` to adjust it for the needs of the new version. +older and the newer kernel got built with a similar configuration. This can be +achieved by using ``make olddefconfig``, as explained in more detail by +Documentation/admin-guide/reporting-regressions.rst; that document also +provides a good deal of other information about regressions you might want to be +aware of. Write and send the report
@@ -1283,7 +1269,7 @@ them when sending the report by mail. If you filed it in a bug tracker, forward the report's text to these addresses; but on top of it put a small note where you mention that you filed it with a link to the ticket. -See 'Documentation/admin-guide/security-bugs.rst' for more information. +See Documentation/admin-guide/security-bugs.rst for more information. Duties after the report went out
@@ -1571,7 +1557,7 @@ Once your report is out your might get asked to do a proper one, as it allows to pinpoint the exact change that causes the issue (which then can easily get reverted to fix the issue quickly). Hence consider to do a proper bisection right away if time permits. See the section 'Special care for regressions' and -the document 'Documentation/admin-guide/bug-bisect.rst' for details how to +the document Documentation/admin-guide/bug-bisect.rst for details how to perform one. In case of a successful bisection add the author of the culprit to the recipients; also CC everyone in the signed-off-by chain, which you find at the end of its commit message.
@@ -1594,7 +1580,7 @@ Some fixes are too complex Even small and seemingly obvious code-changes sometimes introduce new and totally unexpected problems. The maintainers of the stable and longterm kernels are very aware of that and thus only apply changes to these kernels that are -within rules outlined in 'Documentation/process/stable-kernel-rules.rst'. +within rules outlined in Documentation/process/stable-kernel-rules.rst. Complex or risky changes for example do not qualify and thus only get applied to mainline. Other fixes are easy to get backported to the newest stable and
@@ -1756,10 +1742,23 @@ art will lay some groundwork to improve the situation over time. .. - This text is maintained by Thorsten Leemhuis <[email protected]>. If you - spot a typo or small mistake, feel free to let him know directly and he'll - fix it. You are free to do the same in a mostly informal way if you want - to contribute changes to the text, but for copyright reasons please CC + end-of-content +.. + This document is maintained by Thorsten Leemhuis <[email protected]>. If + you spot a typo or small mistake, feel free to let him know directly and + he'll fix it. You are free to do the same in a mostly informal way if you + want to contribute changes to the text, but for copyright reasons please CC [email protected] and "sign-off" your contribution as Documentation/process/submitting-patches.rst outlines in the section "Sign your work - the Developer's Certificate of Origin". +.. + This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top + of the file. If you want to distribute this text under CC-BY-4.0 only, + please use "The Linux kernel developers" for author attribution and link + this as source: + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst +.. + Note: Only the content of this RST file as found in the Linux kernel sources + is available under CC-BY-4.0, as versions of this text that were processed + (for example by the kernel's build system) might contain content taken from + files which use a more restrictive license.
Signature · 1 line
2.35.1
Jonathan Corbet
maintainerQuoted 9 lines · Thorsten Leemhuis
"We don't cause regressions" might be the first rule of Linux kernel development, but it and other aspects of regressions nevertheless are hardly described in the Linux kernel's documentation. The following patches change this by creating two documents dedicated to the topic.
The second patch could easily be folded into the first one, but was kept separate, as it might be a bit controversial. This also allows the patch description to explain some backgrounds for this part of the document. Additionally, ACKs and Reviewed-by tags can be collected separately this way.
Hearing no objections, I have applied this set, thanks.
jon
Thorsten Leemhuis
maintainerQuoted 7 lines · Jonathan Corbet
Quoted 5 lines · Thorsten Leemhuis
"We don't cause regressions" might be the first rule of Linux kernel development, but it and other aspects of regressions nevertheless are hardly described in the Linux kernel's documentation. The following patches change this by creating two documents dedicated to the topic.
Quoted 6 lines
Quoted 4 lines
The second patch could easily be folded into the first one, but was kept separate, as it might be a bit controversial. This also allows the patch description to explain some backgrounds for this part of the document. Additionally, ACKs and Reviewed-by tags can be collected separately this way.
Hearing no objections, I have applied this set, thanks.
Great, thx!
Ciao, Thorsten