How many CI mails is too many?
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.
3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.
Any opinions? Any other ideas?
Signature · 6 lines
Cheers, Arek _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
I feel we generally tend to ignore the results mails for series that we are not actively involved on (although we might be interested in series itself). Also if number of revisions some series can undergo is high, this tendency can grow.
How about the option of sending the results mails to only author and all committers.
Ideally, author should include at lease one committer and in that case we can possibly avoid mail to all committers.
Another option would be no results mails at all and enforce authors to ensure all green at https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updated
Thanks
Sagar
Quoted 26 lines · Arkadiusz Hiler
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.
Quoted 6 lines
3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.Any opinions? Any other ideas?
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 4 lines · Sagar Arun Kamble
I feel we generally tend to ignore the results mails for series that we are not actively involved on (although we might be interested in series itself). Also if number of revisions some series can undergo is high, this tendency can grow.
It is true that I don't care that much about results tied to series I am not interested in, but I don't find this too distracting. They are nested nicely in the thread and are also very easy to distinguish visually.
Quoted 10 lines
How about the option of sending the results mails to only author and all committers. Ideally, author should include at lease one committer and in that case we can possibly avoid mail to all committers.
Another option would be no results mails at all and enforce authors to ensure all green at
https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updatedThanks
Sagar
In my mind the CI system should complement mailing list, not change it or require unnecessary, external interactions. By sending the result to intel-gfx we get the gist of the results here, with the direct links to the patchwork and intel-gfx-ci grids provided (so no need to hunt for those).
The results also stores nicely in the online mailing list archives if we send it to the intel-gfx@fdo.
Searchability and easy categorization is an added bonus if you subscribe to dozen or so mailing lists.
Cheers,
Arek
Quoted 27 lines · Arkadiusz Hiler
Quoted 26 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.
Quoted 5 lines
Quoted 5 lines
3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.Any opinions? Any other ideas?
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 6 lines · Arkadiusz Hiler
Quoted 4 lines · Sagar Arun Kamble
I feel we generally tend to ignore the results mails for series that we are not actively involved on (although we might be interested in series itself). Also if number of revisions some series can undergo is high, this tendency can grow.
Quoted 8 lines
It is true that I don't care that much about results tied to series I am not interested in, but I don't find this too distracting. They are nested nicely in the thread and are also very easy to distinguish visually.
Quoted 3 lines
How about the option of sending the results mails to only author and all committers. Ideally, author should include at lease one committer and in that case we can possibly avoid mail to all committers.
I had the same idea but then I thought that sometimes if I want to apply some series of patches I would like to know what is a status of that patch (fail or not) so maybe sending that to author only can be not really good idea.
-Ewelina
Quoted 54 lines
Quoted 7 lines
Another option would be no results mails at all and enforce authors to ensure all green at
https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updatedThanks
SagarIn my mind the CI system should complement mailing list, not change it or require unnecessary, external interactions. By sending the result to intel-gfx we get the gist of the results here, with the direct links to the patchwork and intel-gfx-ci grids provided (so no need to hunt for those).
The results also stores nicely in the online mailing list archives if we send it to the intel-gfx@fdo.
Searchability and easy categorization is an added bonus if you subscribe to dozen or so mailing lists.
Cheers,
ArekQuoted 31 lines · Arkadiusz Hiler
Quoted 30 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Any opinions? Any other ideas?
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 22 lines · Arkadiusz Hiler
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.
Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.
I'd vote for number 3. And let's see how noisy that gets and maybe that end up in a good thing because we will start seeing what causes noise and possibly work to avoid those.
Quoted 10 lines
Any opinions? Any other ideas?
Signature · 6 lines
Cheers, Arek _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 22 lines · Arkadiusz Hiler
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.
Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.
The best user experience I could think of;
1. If all CI checks succeed, delay and only send one mail with all the results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer gets to work on the fix.
Above requires that all the checks complete rather quickly and a trust is gained to the system so that the absence of e-mail always means the series is doing good, not that the system is clogged in some way :)
Regards, Joonas
Quoted 3 lines
Any opinions? Any other ideas?
Signature · 7 lines
Joonas Lahtinen Open Source Technology Center Intel Corporation _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoting Joonas Lahtinen (2017-11-28 08:15:13)
Quoted 33 lines · Arkadiusz Hiler
Quoted 22 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.The best user experience I could think of;
Quoted 8 lines
1. If all CI checks succeed, delay and only send one mail with all the results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer gets to work on the fix.Above requires that all the checks complete rather quickly and a trust is gained to the system so that the absence of e-mail always means the series is doing good, not that the system is clogged in some way :)
Or just 2. The first being the compilation report; saying we have received your patch and it compiles fine, it will be queued to the farm currently in slot N (or it doesn't even compile!). The second being the success or failure of the CI run.
From the user pov, we can't do anything until the CI report so intermediate emails saying congrats are just fluff. Useful simply to know the patch hasn't fall out of the system, but not supplying any actionable information.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 48 lines · Chris Wilson
Quoting Joonas Lahtinen (2017-11-28 08:15:13)
Quoted 33 lines · Arkadiusz Hiler
Quoted 22 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.The best user experience I could think of;
Quoted 8 lines
1. If all CI checks succeed, delay and only send one mail with all the results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer gets to work on the fix.Above requires that all the checks complete rather quickly and a trust is gained to the system so that the absence of e-mail always means the series is doing good, not that the system is clogged in some way :)
Or just 2. The first being the compilation report; saying we have received your patch and it compiles fine, it will be queued to the farm currently in slot N (or it doesn't even compile!). The second being the success or failure of the CI run.
Quoted 4 lines
From the user pov, we can't do anything until the CI report so intermediate emails saying congrats are just fluff. Useful simply to know the patch hasn't fall out of the system, but not supplying any actionable information.
BAT was meant to be that mail, with the added benefit that if a series fails the basic sanity check you can ignore it for review and everything. Still not quite there yet (and the recently undone change of ratelimiting didn't help).
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoting Daniel Vetter (2017-11-28 10:08:56)
Quoted 54 lines · Chris Wilson
Quoted 48 lines
Quoting Joonas Lahtinen (2017-11-28 08:15:13)
Quoted 33 lines · Arkadiusz Hiler
Quoted 22 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.The best user experience I could think of;
Quoted 8 lines
1. If all CI checks succeed, delay and only send one mail with all the results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer gets to work on the fix.Above requires that all the checks complete rather quickly and a trust is gained to the system so that the absence of e-mail always means the series is doing good, not that the system is clogged in some way :)
Or just 2. The first being the compilation report; saying we have received your patch and it compiles fine, it will be queued to the farm currently in slot N (or it doesn't even compile!). The second being the success or failure of the CI run.
Quoted 4 lines
From the user pov, we can't do anything until the CI report so intermediate emails saying congrats are just fluff. Useful simply to know the patch hasn't fall out of the system, but not supplying any actionable information.
Quoted 4 lines
BAT was meant to be that mail, with the added benefit that if a series fails the basic sanity check you can ignore it for review and everything. Still not quite there yet (and the recently undone change of ratelimiting didn't help).
The compile check should take 30s(?) on the build host with all the distcc/ccache. It's going to be rare to develop a long queue and significant latency; whereas one developer can flood the system with 5 different series they happen to have queued, repeat for everyone getting to work in the morning.
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoting Chris Wilson (2017-11-28 10:16:17)
Quoted 60 lines
Quoting Daniel Vetter (2017-11-28 10:08:56)
Quoted 54 lines · Chris Wilson
Quoted 48 lines
Quoting Joonas Lahtinen (2017-11-28 08:15:13)
Quoted 33 lines · Arkadiusz Hiler
Quoted 22 lines
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.The best user experience I could think of;
Quoted 8 lines
1. If all CI checks succeed, delay and only send one mail with all the results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer gets to work on the fix.Above requires that all the checks complete rather quickly and a trust is gained to the system so that the absence of e-mail always means the series is doing good, not that the system is clogged in some way :)
Or just 2. The first being the compilation report; saying we have received your patch and it compiles fine, it will be queued to the farm currently in slot N (or it doesn't even compile!). The second being the success or failure of the CI run.
Quoted 4 lines
From the user pov, we can't do anything until the CI report so intermediate emails saying congrats are just fluff. Useful simply to know the patch hasn't fall out of the system, but not supplying any actionable information.
Quoted 4 lines
BAT was meant to be that mail, with the added benefit that if a series fails the basic sanity check you can ignore it for review and everything. Still not quite there yet (and the recently undone change of ratelimiting didn't help).
Quoted 5 lines
The compile check should take 30s(?) on the build host with all the distcc/ccache. It's going to be rare to develop a long queue and significant latency; whereas one developer can flood the system with 5 different series they happen to have queued, repeat for everyone getting to work in the morning.
One thing I forgot to mention, is that trybot has a large latency for that single BAT success email. Having a quick "patch received; compiles" response from the system would give a nice bit of reassurance. -Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 22 lines · Arkadiusz Hiler
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.
Quoted 7 lines
2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.
Option 3 sounds good to me.
I don't see it as noisy - if something fails a dedicated notification is in order, and since it gets assigned into a respective thread by the MUA, the noise is minimal. And you only see self-inflicted noise, unless you don't use threading, or are keeping all the threads expanded, which is already non-workable to start with.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 26 lines · Arkadiusz Hiler
Hey all,
For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT results and "full IGT" results (if BAT has not failed).
Recently we have added 32bit build check, and if that fails it sends out additional mail In-Reply-To the series.
I am working on adding some static checks to the CI (spare and checkpatch at the moment, more may come in the future), which may generate even more commotion on the mailing list.
How much of CI noise is too much and how you would like to have the results grouped?
Couple of options to start the discussion:
1. Group all static checks (and the 32bit build?) into one mail:
- just one additional mail,
- may be hard to read in case of catastrophic failure,
- we can send it only when something actually fails.2. Send out the results as a part of BAT results:
- even less noise than (1),
- BAT results already feel cluttered, this may decrease readability.
Quoted 5 lines
3. Have each check as a separate mail, but send it only if the check fails:
- noisy: may result in many mails, depending how many checks fail,
- easier to read and easier to follow on patchwork.Any opinions? Any other ideas?
All static checks grouped, and only sent out when there's something iffy. I think for BAT&functional tests positive confirmation from CI is good, but for static checkers it's a bit more meh imo. So 1&3 for me.
And yes I'd group all the static checks and build checks into 1. E.g. on dri-devel we also want to build-test arm and arm64 using the drm-misc configs.
-Daniel
Quoted 8 lines
--
Cheers,
Arek
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx