Public i915 CI shardruns are disabled
Subsystems (2)
Hello,
The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:
fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1
More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue
I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded
The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.
I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.
Regards,
Tomi Sarvela
--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
More information (excuse my top-posting):
- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.
- Filesystem trashed is the one containing swapfile
- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile
- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better
Regards,
Tomi Sarvela
From: Sarvela, Tomi P
Sent: Tuesday, March 2, 2021 1:38 PM
To: [email protected]
Cc: Szwichtenberg, Radoslaw <[email protected]>
Subject: Public i915 CI shardruns are disabled
Hello,
The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:
fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1
More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue
I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded
The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.
I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.
Regards,
Tomi Sarvela
--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
The regression has been identified; Chris Wilson found commits touching swapfile.c, and reverting them the issue couldn't be reproduced any more.
https://patchwork.freedesktop.org/series/87549/
This revert will be applied to core-for-CI branch. When new CI_DRM has been built, shard-testing will be enabled again.
Regards,
Tomi Sarvela
From: Sarvela, Tomi P
More information (excuse my top-posting):
- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.
- Filesystem trashed is the one containing swapfile
- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile
- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better
Regards,
Tomi Sarvela
From: Sarvela, Tomi P
Hello,
The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:
fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1
More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue
I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded
The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.
I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.
Regards,
Tomi Sarvela
--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 7 lines · Tomi P
The regression has been identified; Chris Wilson found commits touching
swapfile.c, and reverting them the issue couldn’t be reproduced any more.
Quoted 7 lines
https://patchwork.freedesktop.org/series/87549/
This revert will be applied to core-for-CI branch. When new
CI_DRMhasbeen built, shard-testing will be enabled again.
Just making sure this is on the radar upstream.
Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Adding the right people.
It seems that the three commits that needed reverting are
f885056a48cc ("mm: simplify swapdev_block")
3e3126cf2a6d ("mm: only make map_swap_entry available for CONFIG_HIBERNATION") 48d15436fde6 ("mm: remove get_swap_bio")
and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.
I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.
Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.
Jens, Christoph?
Linus
Quoted 13 lines · Dave Airlie
Quoted 11 lines · Tomi P
The regression has been identified; Chris Wilson found commits touching
swapfile.c, and reverting them the issue couldn’t be reproduced any more.
Quoted 7 lines
Quoted 3 lines
This revert will be applied to core-for-CI branch. When new
CI_DRMhasbeen built, shard-testing will be enabled again.
Just making sure this is on the radar upstream.
Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 12 lines · Linus Torvalds
Adding the right people.
It seems that the three commits that needed reverting are
f885056a48cc("mm: simplifyswapdev_block")
3e3126cf2a6d("mm: only makemap_swap_entryavailable forCONFIG_HIBERNATION")48d15436fde6("mm: removeget_swap_bio")and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.
Quoted 8 lines
I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.
Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.
Jens, Christoph?
Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.
https://git.kernel.dk/cgit/linux-block/commit/?h=swap-fix
Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 5 lines · Dave Airlie
Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.
Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.
Jens, if that can get confirmed, please send it my way asap.. Thanks,
Linus
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Dave Airlie
Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.
Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.
Jens, if that can get confirmed, please send it my way asap.. Thanks,
Yep, it's the same issue indeed. Was made aware of it after lunch today and emailed Christoph, but then decided to dig into it myself a few hours later. Andrew already queued it up I just saw, but I noticed that that version will break on !CONFIG_HIBERNATION.
Patch below if you just want to grab it.
commit e25b1010db005a59727e1ff5f43af889effd31a3
Author: Jens Axboe <[email protected]> Date: Tue Mar 2 14:53:21 2021 -0700
swap: fix swapfile read/write offset
We're not factoring in the start of the file for where to write and read the swapfile, which leads to very unfortunate side effects of writing where we should not be...
include/linux/swap.h +1 -0
@@ -485,6 +485,7 @@ struct backing_dev_info;extern int init_swap_address_space (unsigned int type,unsigned long nr_pages);extern void exit_swap_address_space (unsigned int type);extern struct swap_info_struct *get_swap_device (swp_entry_t entry); +sector_t swap_page_sector (struct page *page);static inline void put_swap_device (struct swap_info_struct *si) {
mm/page_io.c +0 -5
@@ -254,11 +254,6 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)return ret; } -static sector_t swap_page_sector (struct page *page) -{ -return (sector_t )__page_file_index (page) << (PAGE_SHIFT -9 ); -} -static inline void count_swpout_vm_event (struct page *page) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/swapfile.c +13 -0
@@ -219,6 +219,19 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)BUG (); } +sector_t swap_page_sector (struct page *page) +{ +struct swap_info_struct *sis =page_swap_info (page); +struct swap_extent *se; +sector_t sector; +pgoff_t offset; + + offset =__page_file_index (page); + se =offset_to_swap_extent (sis, offset); + sector = se->start_block + (offset - se->start_page); +return sector << (PAGE_SHIFT -9 ); +} +/* * swap allocation tell device that a cluster of swap can now be discarded, * to allow the swap device to optimize its wear-levelling.
Signature · 6 lines
Jens Axboe _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Dave Airlie
Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.
Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.
Quoted 8 lines
Quoted 1 line
Jens, if that can get confirmed, please send it my way asap.. Thanks,
Yep, it's the same issue indeed. Was made aware of it after lunch today and emailed Christoph, but then decided to dig into it myself a few hours later. Andrew already queued it up I just saw, but I noticed that that version will break on !CONFIG_HIBERNATION.
Patch below if you just want to grab it.
Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.
Signature · 6 lines
Jens Axboe _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 3 lines · Jens Axboe
Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.
Ok, if you have something else pending anyway, let's do that. Send me the pull request, and I'll take it asap.
Thanks,
Linus
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.
Ok, if you have something else pending anyway, let's do that. Send me the pull request, and I'll take it asap.
Done
Signature · 6 lines
Jens Axboe _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Ok, slightly delayed by dinner, but commit caf6912f3f4a ("swap: fix swapfile read/write offset") is out in my tree now.
Dave - can you check that the current -git works for your CI people?
Thanks,
Linus
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.
Quoted 8 lines
Quoted 2 lines
Ok, if you have something else pending anyway, let's do that. Send me the pull request, and I'll take it asap.
Done
--
Jens Axboe
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
From my earlier message on the mailing list:
[...] "Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will."
The CI public interface doesn't really show this: the hosts started testing, died, and in boot stuck to the grub menu because grub.cfg (or anything) wasn't available on root disk.
Decision to shut down the extended testing was mine, when I saw ~1 host per shard dying each testing round (couple of hosts per hour).
It's a kind of bug our CI is not handling well, because on the catastrophic scale the effects are close to the maximum (where max would be permanent hw damage), and cause is not related to i915 at all.
Regards,
Tomi Sarvela
Quoted 58 lines
From: Linus Torvalds <[email protected]>
Sent: Wednesday, March 3, 2021 1:28 AM
To: Dave Airlie <[email protected]>; Jens Axboe <[email protected]>; Christoph Hellwig <[email protected]>; Damien Le Moal <[email protected]>; Johannes Thumshirn <[email protected]>; Chaitanya Kulkarni <[email protected]>
Cc: Sarvela, Tomi P <[email protected]>; Linux Memory Management List <[email protected]>; Andrew Morton <[email protected]>; [email protected]
Subject: Re: [Intel-gfx] Public i915 CI shardruns are disabledAdding the right people.
It seems that the three commits that needed reverting are
f885056a48cc("mm: simplifyswapdev_block")3e3126cf2a6d("mm: only makemap_swap_entryavailable forCONFIG_HIBERNATION")
48d15436fde6("mm: removeget_swap_bio")and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.
I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.
Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.
Jens, Christoph?
Linus
Quoted 2 lines · Dave Airlie
On Wed, 3 Mar 2021 at 03:27, Sarvela, Tomi P <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
The regression has been identified; Chris Wilson found commits touching
swapfile.c, and reverting them the issue couldn’t be reproduced anymore.
Quoted 14 lines
Quoted 10 lines
https://patchwork.freedesktop.org/series/87549/
This revert will be applied to core-for-CI branch. When new
CI_DRMhasbeen built, shard-testing will be enabled again.
Just making sure this is on the radar upstream.
Dave.
Patchwork
CI[Intel-gfx] ✗ Fi.CI.BAT: failure for Public i915 CI shardruns are disabled
In reply to Sarvela, Tomi P== Series Details ==
== Series Details ==
Series: Public i915 CI shardruns are disabled
URL : https://patchwork.freedesktop.org/series/87558/
State : failure
== Summary ==
Applying: Public i915 CI shardruns are disabled
Using index info to reconstruct a base tree...
M include/linux/swap.h
M mm/page_io.c
M mm/swapfile.c
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
It seems that the chops to be built has been re-defined several times in pipelines. Fixed.
Tomi
From: Sarvela, Tomi P
Sent: Tuesday, March 2, 2021 7:27 PM
To: '[email protected]' <[email protected]>
Cc: Szwichtenberg, Radoslaw <[email protected]>
Subject: RE: Public i915 CI shardruns are disabled
The regression has been identified; Chris Wilson found commits touching swapfile.c, and reverting them the issue couldn't be reproduced any more.
https://patchwork.freedesktop.org/series/87549/
This revert will be applied to core-for-CI branch. When new CI_DRM has been built, shard-testing will be enabled again.
Regards,
Tomi Sarvela
From: Sarvela, Tomi P
More information (excuse my top-posting):
- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.
- Filesystem trashed is the one containing swapfile
- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile
- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better
Regards,
Tomi Sarvela
From: Sarvela, Tomi P
Hello,
The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:
fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1
More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue
I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded
The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.
I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.
Regards,
Tomi Sarvela
--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx