Add goals of patch review and tips
PATCH v2 1/1 Add goals of patch review and tips
I see two options for patch review. Either we have a single senior maintainer who does most of or it is distributed.
For now I think it needs to be distributed which is beyond the scope of this commit.
In order to distribute it we need new contributors to review each others' work at least for the first few revisions.
I think that anyone can review a patch if they put the work in to test it and try to break it. Then understand why it is broken.
This commit states some ideas about how to do that, plus some tips for more advanced patch review.
---
I'd like to clear this off the queue now.
V2:
* Correct typo
* Use Einstein quote
* Mention Tested-by
doc/maintainer-patch-review-checklist.txt | 82 ++++++++++++++++++++++- 1 file changed, 81 insertions(+), 1 deletion(-)
doc/maintainer-patch-review-checklist.txt +81 -1
@@ -1,4 +1,84 @@ -# Maintainer Patch Review Checklist +# Patch Review + +Anyone can and should review patches. It's the only way to get good at +patch review and for the project to scale. + +## Goals of patch review + +1. Prevent false positive test results +2. Prevent false negative test results +3. Keep the code as simple as possible, but no simpler + +## How to find clear errors + +A clear error is one where there is unlikely to be any argument if you +provide evidence of it. Evidence being an error trace or logical proof +that an error will occur in a common situation. + +The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master. +* Compile the patch with default and non-default configurations. + - Use sanitizers e.g. undefined behaviour, address. + - Compile on non-x86 + - Compile on x86 with -m32 +* Use `make check` +* Run effected tests in a VM + - Use single vCPU + - Use many vCPUs and enable NUMA + - Restrict RAM to < 1GB. +* Run effected tests on an embedded device +* Run effected tests on non-x86 machine in general +* Run reproducers on a kernel where the bug is present +* Run tests with "-i0" +* Compare usage of system calls with man page descriptions +* Compare usage of system calls with kernel code +* Search the LTP library for existing helper functions + +## How to find subtle errors + +A subtle error is one where you can expect some argument because you +do not have clear evidence of an error. It is best to state these as +questions and not make assertions if possible. + +Although if it is a matter of style or "taste" then senior maintainers +can assert what is correct to avoid bike shedding. + +* Ask what happens if there is an error, could it be debugged just + with the test output? +* Are we testing undefined behavior? + - Could future kernel behaviour change without "breaking userland"? + - Does the kernel behave differently depending on hardware? + - Does it behave differently depending kernel on configuration? + - Does it behave differently depending on the compiler? +* Will it scale to tiny and huge systems? + - What happens if there are 100+ CPUs? + - What happens if each CPU core is very slow? + - What happens if there are 2TB or RAM? +* Are we repeating a pattern that can be turned into a library function? +* Is a single test trying to do too much? +* Could multiple similar tests be merged? +* Race conditions + - What happens if a process gets preempted? + - Could checkpoints or fuzzsync by used instead? + - Note, usually you can insert a sleep to prove a race condition + exists however finding them is hard +* Is there a simpler way to achieve the same kernel coverage? + +## How to get patches merged + +Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers Checklist Patchset should be tested locally and ideally also in maintainer's fork in GitHub Actions on GitHub.
Signature · 5 lines
2.41.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Richie,
Thank you for this, I'll also start investing more effort in review.
with few nits below.
Quoted 79 lines · IST Richard Palethorpe via ltp
I see two options for patch review. Either we have a single senior maintainer who does most of or it is distributed.
For now I think it needs to be distributed which is beyond the scope of this commit.
In order to distribute it we need new contributors to review each others' work at least for the first few revisions.
I think that anyone can review a patch if they put the work in to test it and try to break it. Then understand why it is broken.
This commit states some ideas about how to do that, plus some tips for more advanced patch review.
Signed-off-by Richard Palethorpe <[email protected]>Cc Cyril Hrubis <[email protected]>---
I'd like to clear this off the queue now.
V2:
* Correct typo
* Use Einstein quote
* Mention Tested-bydoc/maintainer-patch-review-checklist.txt | 82 ++++++++++++++++++++++- 1 file changed, 81 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/doc/maintainer-patch-review-checklist.txtindex61eb06c5f..b11c7b546100644doc/maintainer-patch-review-checklist.txt +42
-1@@ -1,4 +1,84 @@ -# Maintainer Patch Review Checklist +# Patch Review + +Anyone can and should review patches. It's the only way to get good at +patch review and for the project to scale. + +## Goals of patch review + +1. Prevent false positive test results +2. Prevent false negative test results +3. Keep the code as simple as possible, but no simpler + +## How to find clear errors + +A clear error is one where there is unlikely to be any argument if you +provide evidence of it. Evidence being an error trace or logical proof +that an error will occur in a common situation. + +The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master. +* Compile the patch with default and non-default configurations. + - Use sanitizers e.g. undefined behaviour, address. + - Compile on non-x86 + - Compile on x86 with -m32 +* Use `make check` +* Run effected tests in a VM + - Use single vCPU + - Use many vCPUs and enable NUMA + - Restrict RAM to < 1GB. +* Run effected tests on an embedded device +* Run effected tests on non-x86 machine in general +* Run reproducers on a kernel where the bug is present +* Run tests with "-i0" +* Compare usage of system calls with man page descriptions +* Compare usage of system calls with kernel code +* Search the LTP library for existing helper functions + +## How to find subtle errors + +A subtle error is one where you can expect some argument because you +do not have clear evidence of an error. It is best to state these as +questions and not make assertions if possible.
doc/maintainer-patch-review-checklist.txt line 52
Quoted 10 lines
doc/maintainer-patch-review-checklist.txt +10
-0+ +Although if it is a matter of style or "taste" then senior maintainers +can assert what is correct to avoid bike shedding. + +* Ask what happens if there is an error, could it be debugged just + with the test output? +* Are we testing undefined behavior? + - Could future kernel behaviour change without "breaking userland"? + - Does the kernel behave differently depending on hardware? + - Does it behave differently depending kernel on configuration?
depending on kernel configuration
doc/maintainer-patch-review-checklist.txt line 57
Quoted 5 lines
doc/maintainer-patch-review-checklist.txt +5
-0+ - Does it behave differently depending on the compiler? +* Will it scale to tiny and huge systems? + - What happens if there are 100+ CPUs? + - What happens if each CPU core is very slow? + - What happens if there are 2TB or RAM?
s/or/of
Quoted 27 lines
doc/maintainer-patch-review-checklist.txt +24
-0+* Are we repeating a pattern that can be turned into a library function? +* Is a single test trying to do too much? +* Could multiple similar tests be merged? +* Race conditions + - What happens if a process gets preempted? + - Could checkpoints or fuzzsync by used instead? + - Note, usually you can insert a sleep to prove a race condition + exists however finding them is hard +* Is there a simpler way to achieve the same kernel coverage? + +## How to get patches merged + +Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers Checklist Patchset should be tested locally and ideally also in maintainer's fork in GitHub Actions on GitHub.
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 49 lines
doc/maintainer-patch-review-checklist.txt +43
-1@@ -1,4 +1,84 @@ -# Maintainer Patch Review Checklist +# Patch Review + +Anyone can and should review patches. It's the only way to get good at +patch review and for the project to scale. + +## Goals of patch review + +1. Prevent false positive test results +2. Prevent false negative test results +3. Keep the code as simple as possible, but no simpler + +## How to find clear errors + +A clear error is one where there is unlikely to be any argument if you +provide evidence of it. Evidence being an error trace or logical proof +that an error will occur in a common situation. + +The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master. +* Compile the patch with default and non-default configurations. + - Use sanitizers e.g. undefined behaviour, address. + - Compile on non-x86 + - Compile on x86 with -m32 +* Use `make check` +* Run effected tests in a VM + - Use single vCPU + - Use many vCPUs and enable NUMA + - Restrict RAM to < 1GB. +* Run effected tests on an embedded device +* Run effected tests on non-x86 machine in general +* Run reproducers on a kernel where the bug is present +* Run tests with "-i0" +* Compare usage of system calls with man page descriptions +* Compare usage of system calls with kernel code +* Search the LTP library for existing helper functions + +## How to find subtle errors + +A subtle error is one where you can expect some argument because you +do not have clear evidence of an error. It is best to state these as +questions and not make assertions if possible. +
doc/maintainer-patch-review-checklist.txt line 53
Quoted 10 lines
doc/maintainer-patch-review-checklist.txt +10
-0+Although if it is a matter of style or "taste" then senior maintainers +can assert what is correct to avoid bike shedding. + +* Ask what happens if there is an error, could it be debugged just + with the test output? +* Are we testing undefined behavior? + - Could future kernel behaviour change without "breaking userland"? + - Does the kernel behave differently depending on hardware? + - Does it behave differently depending kernel on configuration? + - Does it behave differently depending on the compiler?
- Does it behave differently when order of checks on syscall parameters change in kernel?
We used to have quite some tests that passed two or more invalid parameters to a sysycall expecting one of them would be checked first...
Quoted 18 lines
doc/maintainer-patch-review-checklist.txt +18
-0+* Will it scale to tiny and huge systems? + - What happens if there are 100+ CPUs? + - What happens if each CPU core is very slow? + - What happens if there are 2TB or RAM? +* Are we repeating a pattern that can be turned into a library function? +* Is a single test trying to do too much? +* Could multiple similar tests be merged? +* Race conditions + - What happens if a process gets preempted? + - Could checkpoints or fuzzsync by used instead? + - Note, usually you can insert a sleep to prove a race condition + exists however finding them is hard +* Is there a simpler way to achieve the same kernel coverage? + +## How to get patches merged + +Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting
doc/maintainer-patch-review-checklist.txt line 81
Quoted 10 lines
doc/maintainer-patch-review-checklist.txt +10
-0+the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers Checklist
Looks very nice, thanks for writing this out.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 18 lines · Cyril Hrubis
Quoted 18 lines
doc/maintainer-patch-review-checklist.txt +12
-1@@ -1,4 +1,84 @@ -# Maintainer Patch Review Checklist +# Patch Review + +Anyone can and should review patches. It's the only way to get good at +patch review and for the project to scale. + +## Goals of patch review + +1. Prevent false positive test results +2. Prevent false negative test results +3. Keep the code as simple as possible, but no simpler + +## How to find clear errors
Quoted 8 lines
doc/maintainer-patch-review-checklist.txt line 20
Quoted 8 lines
doc/maintainer-patch-review-checklist.txt +8
-0+ +A clear error is one where there is unlikely to be any argument if you +provide evidence of it. Evidence being an error trace or logical proof +that an error will occur in a common situation. + +The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master.
As a newbie with LTP I am still struggling to understand some things like this one. How is possible to merge to master in order to review?
Quoted 39 lines
Quoted 23 lines
doc/maintainer-patch-review-checklist.txt +23
-0+* Compile the patch with default and non-default configurations. + - Use sanitizers e.g. undefined behaviour, address. + - Compile on non-x86 + - Compile on x86 with -m32 +* Use `make check` +* Run effected tests in a VM + - Use single vCPU + - Use many vCPUs and enable NUMA + - Restrict RAM to < 1GB. +* Run effected tests on an embedded device +* Run effected tests on non-x86 machine in general +* Run reproducers on a kernel where the bug is present +* Run tests with "-i0" +* Compare usage of system calls with man page descriptions +* Compare usage of system calls with kernel code +* Search the LTP library for existing helper functions + +## How to find subtle errors + +A subtle error is one where you can expect some argument because you +do not have clear evidence of an error. It is best to state these as +questions and not make assertions if possible. +doc/maintainer-patch-review-checklist.txt line 53
Quoted 10 lines
doc/maintainer-patch-review-checklist.txt +10
-0+Although if it is a matter of style or "taste" then senior maintainers +can assert what is correct to avoid bike shedding. + +* Ask what happens if there is an error, could it be debugged just + with the test output? +* Are we testing undefined behavior? + - Could future kernel behaviour change without "breaking userland"? + - Does the kernel behave differently depending on hardware? + - Does it behave differently depending kernel on configuration? + - Does it behave differently depending on the compiler?- Does it behave differently when order of checks on syscall parameters change in kernel?
We used to have quite some tests that passed two or more invalid parameters to a sysycall expecting one of them would be checked first...
Quoted 15 lines
Quoted 5 lines
doc/maintainer-patch-review-checklist.txt +5
-0+* Will it scale to tiny and huge systems? + - What happens if there are 100+ CPUs? + - What happens if each CPU core is very slow? + - What happens if there are 2TB or RAM? +* Are we repeating a pattern that can be turned into a library function?doc/maintainer-patch-review-checklist.txt line 68
Quoted 10 lines
doc/maintainer-patch-review-checklist.txt +10
-0+* Is a single test trying to do too much? +* Could multiple similar tests be merged? +* Race conditions + - What happens if a process gets preempted? + - Could checkpoints or fuzzsync by used instead? + - Note, usually you can insert a sleep to prove a race condition + exists however finding them is hard +* Is there a simpler way to achieve the same kernel coverage? + +## How to get patches merged
Again from my POV the description is more about what you should do as a reviewer than how to get a patch merged.
Quoted 16 lines
doc/maintainer-patch-review-checklist.txt line 81
Quoted 13 lines
doc/maintainer-patch-review-checklist.txt +13
-0+ +Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers ChecklistLooks very nice, thanks for writing this out.
Quoted 5 lines
Reviewed-by Cyril Hrubis <[email protected]>Signature · 2 lines
Cyril Hrubis [email protected]
I feel that this is more an overview and reminder of already contributors. Not sure how helpful is it for new comers like myself
Signature · 4 lines
Sent with my mu4e -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 6 lines
Quoted 3 lines
doc/maintainer-patch-review-checklist.txt line 20
Quoted 3 lines
doc/maintainer-patch-review-checklist.txt +3
-0+The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master.As a newbie with LTP I am still struggling to understand some things like this one. How is possible to merge to master in order to review?
Obviously you do that to your local git tree. This is basics of git development nothing specific to LTP here.
Quoted 4 lines
Quoted 1 line
Quoted 1 line
+## How to get patches merged
Again from my POV the description is more about what you should do as a reviewer than how to get a patch merged.
Isn't that the same? If you know what are developers doing in order to catch common mistakes you can as well avoid doing them...
Quoted 15 lines
Quoted 15 lines
doc/maintainer-patch-review-checklist.txt line 81
Quoted 12 lines
doc/maintainer-patch-review-checklist.txt +12
-0+Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers ChecklistLooks very nice, thanks for writing this out.
Quoted 8 lines
Quoted 5 lines
Reviewed-by Cyril Hrubis <[email protected]>Signature · 2 lines
Cyril Hrubis [email protected]I feel that this is more an overview and reminder of already contributors. Not sure how helpful is it for new comers like myself
I think that there are different levels of newcommers. I do not think that the documment is supposed to help newcommers that are already familiar with how git based development works and only highlights things that are specific to LTP.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hello,
Quoted 10 lines · Cyril Hrubis
Hi!
Quoted 6 lines
Quoted 3 lines
doc/maintainer-patch-review-checklist.txt line 20
Quoted 3 lines
doc/maintainer-patch-review-checklist.txt +3
-0+The following are examples and may not be appropriate for all tests. + +* Merge the patch. It should apply cleanly to master.As a newbie with LTP I am still struggling to understand some things like this one. How is possible to merge to master in order to review?
Obviously you do that to your local git tree. This is basics of git development nothing specific to LTP here.
I suppose it would not hurt to add "to your local copy of the master branch". It's only a few more words that would clean it up (IMO). If that doesn't make sense then you will have to do a search or ask someone because this document can't be too long.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
+## How to get patches merged
Again from my POV the description is more about what you should do as a reviewer than how to get a patch merged.
Isn't that the same? If you know what are developers doing in order to catch common mistakes you can as well avoid doing them...
Perhaps we are not doing a good job of marketing patch review in this document, but it is probably also outside the scope of this document.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
doc/maintainer-patch-review-checklist.txt line 81
Quoted 12 lines
doc/maintainer-patch-review-checklist.txt +12
-0+Once you think a patch is good enough you should add your Reviewed-by +and/or Tested-by tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the Tested-by tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +## Maintainers ChecklistLooks very nice, thanks for writing this out.
Reviewed-by Cyril Hrubis <[email protected]>Signature · 2 lines
Cyril Hrubis [email protected]
Quoted 7 lines
Quoted 2 lines
I feel that this is more an overview and reminder of already contributors. Not sure how helpful is it for new comers like myself
I think that there are different levels of newcommers. I do not think that the documment is supposed to help newcommers that are already familiar with how git based development works and only highlights things that are specific to LTP.
Yup, there is a long tutorial which explains in depth a lot of stuff and this could be expanded, but I don't have time for that right now.
Signature · 5 lines
Thank you, Richard. -- Mailing list info: https://lists.linux.it/listinfo/ltp