Skip to contentSkip to conversation

Public i915 CI shardruns are disabled

Intel-gfx Sarvela, Tomi P 5 participants 15 messages Mar 2, 2021 – Mar 9, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:

fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1

More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue

I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded

The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.

I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.

Regards,

Tomi Sarvela

--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#1 lore raw

More information (excuse my top-posting):

- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.

- Filesystem trashed is the one containing swapfile

- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile

- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better

Regards,

Tomi Sarvela

From: Sarvela, Tomi P
Sent: Tuesday, March 2, 2021 1:38 PM
To: [email protected]
Cc: Szwichtenberg, Radoslaw <[email protected]>
Subject: Public i915 CI shardruns are disabled

Hello,

The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:

fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1

More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue

I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded

The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.

I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.

Regards,

Tomi Sarvela

--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#2 lore raw

The regression has been identified; Chris Wilson found commits touching swapfile.c, and reverting them the issue couldn't be reproduced any more.

https://patchwork.freedesktop.org/series/87549/

This revert will be applied to core-for-CI branch. When new CI_DRM has been built, shard-testing will be enabled again.

Regards,

Tomi Sarvela

From: Sarvela, Tomi P

More information (excuse my top-posting):

- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.

- Filesystem trashed is the one containing swapfile

- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile

- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better

Regards,

Tomi Sarvela

From: Sarvela, Tomi P

Hello,

The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:

fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1

More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue

I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded

The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.

I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.

Regards,

Tomi Sarvela

--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw
Quoted 7 lines · Tomi P

The regression has been identified; Chris Wilson found commits touching

swapfile.c, and reverting them the issue couldn’t be reproduced any more.

Quoted 7 lines

https://patchwork.freedesktop.org/series/87549/

This revert will be applied to core-for-CI branch. When new CI_DRM has

been built, shard-testing will be enabled again.

Just making sure this is on the radar upstream.

Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw

Adding the right people.

It seems that the three commits that needed reverting are

f885056a48cc ("mm: simplify swapdev_block")
3e3126cf2a6d ("mm: only make map_swap_entry available for CONFIG_HIBERNATION") 48d15436fde6 ("mm: remove get_swap_bio")

and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.

I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.

Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.

Jens, Christoph?

Linus

Quoted 13 lines · Dave Airlie
Quoted 11 lines · Tomi P

The regression has been identified; Chris Wilson found commits touching

swapfile.c, and reverting them the issue couldn’t be reproduced any more.

https://patchwork.freedesktop.org/series/87549/

Quoted 7 lines
Quoted 3 lines

This revert will be applied to core-for-CI branch. When new CI_DRM has

been built, shard-testing will be enabled again.

Just making sure this is on the radar upstream.

Dave.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw
Quoted 12 lines · Linus Torvalds

Adding the right people.

It seems that the three commits that needed reverting are

f885056a48cc ("mm: simplify swapdev_block")
3e3126cf2a6d ("mm: only make map_swap_entry available for CONFIG_HIBERNATION") 48d15436fde6 ("mm: remove get_swap_bio")

and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.

Quoted 8 lines

I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.

Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.

Jens, Christoph?

Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.

https://git.kernel.dk/cgit/linux-block/commit/?h=swap-fix

Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
Quoted 5 lines · Dave Airlie

Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.

https://git.kernel.dk/cgit/linux-block/commit/?h=swap-fix

Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.

Jens, if that can get confirmed, please send it my way asap.. Thanks,

Linus
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#7 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Dave Airlie

Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.

https://git.kernel.dk/cgit/linux-block/commit/?h=swap-fix

Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.

Jens, if that can get confirmed, please send it my way asap.. Thanks,

Yep, it's the same issue indeed. Was made aware of it after lunch today and emailed Christoph, but then decided to dig into it myself a few hours later. Andrew already queued it up I just saw, but I noticed that that version will break on !CONFIG_HIBERNATION.

Patch below if you just want to grab it.

commit e25b1010db005a59727e1ff5f43af889effd31a3

Author: Jens Axboe <[email protected]>
Date:   Tue Mar 2 14:53:21 2021 -0700

swap: fix swapfile read/write offset

We're not factoring in the start of the file for where to write and read the swapfile, which leads to very unfortunate side effects of writing where we should not be...

Fixes 48d15436fde6 ("mm: remove get_swap_bio")
include/linux/swap.h +1 -0
index 32f665b1ee85..4cc6ec3bf0ab 100644
@@ -485,6 +485,7 @@ struct backing_dev_info;
 extern int init_swap_address_space(unsigned int type, unsigned long nr_pages);
 extern void exit_swap_address_space(unsigned int type);
 extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
+sector_t swap_page_sector(struct page *page);
 
 static inline void put_swap_device(struct swap_info_struct *si)
 {
mm/page_io.c +0 -5
index 485fa5cca4a2..c493ce9ebcf5 100644
@@ -254,11 +254,6 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 	return ret;
 }
 
-static sector_t swap_page_sector(struct page *page)
-{
-	return (sector_t)__page_file_index(page) << (PAGE_SHIFT - 9);
-}
-
 static inline void count_swpout_vm_event(struct page *page)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/swapfile.c +13 -0
index f039745989d2..084a5b9a18e5 100644
@@ -219,6 +219,19 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
 	BUG();
 }
 
+sector_t swap_page_sector(struct page *page)
+{
+	struct swap_info_struct *sis = page_swap_info(page);
+	struct swap_extent *se;
+	sector_t sector;
+	pgoff_t offset;
+
+	offset = __page_file_index(page);
+	se = offset_to_swap_extent(sis, offset);
+	sector = se->start_block + (offset - se->start_page);
+	return sector << (PAGE_SHIFT - 9);
+}
+
 /*
  * swap allocation tell device that a cluster of swap can now be discarded,
  * to allow the swap device to optimize its wear-levelling.
Signature · 6 lines
Jens Axboe

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#8 lore raw
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Dave Airlie

Looks like Jens saw it at least, he posted this on twitter a few mins ago so I assume it'll be incoming soon.

https://git.kernel.dk/cgit/linux-block/commit/?h=swap-fix

Ahh. You use a swap file. This might be the same thing that I think the phoronix people hit as ext4 corruption this merge window.

Quoted 8 lines
Quoted 1 line

Jens, if that can get confirmed, please send it my way asap.. Thanks,

Yep, it's the same issue indeed. Was made aware of it after lunch today and emailed Christoph, but then decided to dig into it myself a few hours later. Andrew already queued it up I just saw, but I noticed that that version will break on !CONFIG_HIBERNATION.

Patch below if you just want to grab it.

Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.

Signature · 6 lines
Jens Axboe

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#10 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe

Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.

Ok, if you have something else pending anyway, let's do that. Send me the pull request, and I'll take it asap.

Done

Signature · 6 lines
Jens Axboe

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#11 lore raw

Ok, slightly delayed by dinner, but commit caf6912f3f4a ("swap: fix swapfile read/write offset") is out in my tree now.

Dave - can you check that the current -git works for your CI people?

Thanks,
Linus

Quoted 7 lines · Jens Axboe
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe

Or if you want a pull, just let me know. Have another misc patch to flush out anyway that doesn't belong in any of my usual branches.

Quoted 8 lines
Quoted 2 lines

Ok, if you have something else pending anyway, let's do that. Send me the pull request, and I'll take it asap.

Done

--
Jens Axboe

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#12 lore raw
In reply to Linus Torvalds

From my earlier message on the mailing list:
[...] "Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will."

The CI public interface doesn't really show this: the hosts started testing, died, and in boot stuck to the grub menu because grub.cfg (or anything) wasn't available on root disk.

Decision to shut down the extended testing was mine, when I saw ~1 host per shard dying each testing round (couple of hosts per hour).

It's a kind of bug our CI is not handling well, because on the catastrophic scale the effects are close to the maximum (where max would be permanent hw damage), and cause is not related to i915 at all.

Regards,

Tomi Sarvela

Quoted 58 lines

From: Linus Torvalds <[email protected]>
Sent: Wednesday, March 3, 2021 1:28 AM
To: Dave Airlie <[email protected]>; Jens Axboe <[email protected]>; Christoph Hellwig <[email protected]>; Damien Le Moal <[email protected]>; Johannes Thumshirn <[email protected]>; Chaitanya Kulkarni <[email protected]>
Cc: Sarvela, Tomi P <[email protected]>; Linux Memory Management List <[email protected]>; Andrew Morton <[email protected]>; [email protected]
Subject: Re: [Intel-gfx] Public i915 CI shardruns are disabled

Adding the right people.

It seems that the three commits that needed reverting are

f885056a48cc ("mm: simplify swapdev_block") 3e3126cf2a6d ("mm: only make map_swap_entry available for
CONFIG_HIBERNATION")
48d15436fde6 ("mm: remove get_swap_bio")

and while they look very harmless to me, let's bring in Christoph and Jens who were actually involved with them.

I'm assuming that it's that third one that is the real issue (and the two other ones were to get to it), but it would also be good to know what the actual details of the regression actually were.

Maybe that's obvious to somebody who has more context about the 9815 CI runs and its web interface, but it sure isn't clear to me.

Jens, Christoph?

Linus

Quoted 2 lines · Dave Airlie

On Wed, 3 Mar 2021 at 03:27, Sarvela, Tomi P <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines

The regression has been identified; Chris Wilson found commits touching

swapfile.c, and reverting them the issue couldn’t be reproduced any

more.

Quoted 14 lines
Quoted 10 lines

https://patchwork.freedesktop.org/series/87549/

This revert will be applied to core-for-CI branch. When new CI_DRM has

been built, shard-testing will be enabled again.

Just making sure this is on the radar upstream.

Dave.

#14 lore raw

[Intel-gfx] ✗ Fi.CI.BAT: failure for Public i915 CI shardruns are disabled

In reply to Sarvela, Tomi P
== Series Details ==

== Series Details ==

Series: Public i915 CI shardruns are disabled
URL : https://patchwork.freedesktop.org/series/87558/
State : failure

== Summary ==

Applying: Public i915 CI shardruns are disabled
Using index info to reconstruct a base tree...
M include/linux/swap.h
M mm/page_io.c
M mm/swapfile.c
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#13 lore raw
In reply to Sarvela, Tomi P

It seems that the chops to be built has been re-defined several times in pipelines. Fixed.

https://github.com/intel-innersource/drivers.gpu.i915.ci.pipelines/commit/89d2f8174a15585c082b2f714551225ba6cafe08

Tomi

From: Sarvela, Tomi P
Sent: Tuesday, March 2, 2021 7:27 PM
To: '[email protected]' <[email protected]>
Cc: Szwichtenberg, Radoslaw <[email protected]>
Subject: RE: Public i915 CI shardruns are disabled

The regression has been identified; Chris Wilson found commits touching swapfile.c, and reverting them the issue couldn't be reproduced any more.

https://patchwork.freedesktop.org/series/87549/

This revert will be applied to core-for-CI branch. When new CI_DRM has been built, shard-testing will be enabled again.

Regards,

Tomi Sarvela

From: Sarvela, Tomi P
More information (excuse my top-posting):

- Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears.

- Filesystem trashed is the one containing swapfile

- If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile

- Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better

Regards,

Tomi Sarvela

From: Sarvela, Tomi P
Hello,

The linux i915 CI shardruns have been disabled. This is due to the unfortunate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance seen in CI_DRM_9818. Last changes coming in were:

fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1

More information can be seen at:
https://phoronix.com/scan.php?page=news_item&px=Linux-5.12-Early-Buggy-Issue

I've seen this bug happen regularly with (but not limited to) IGT test:
igt@gem_tiled_swapping@non-threaded

The range for bisection is linux-next 20210215 to 20210129 because the kernels in-between taint the kernel and our i915 testing was not done. Hitting the bug corrupts the underlying filesystem very thoroughly, wiping out large amount of data from the beginning of the partition which leaves fsck sad with thousands of items lost. Bisection of the IGT testlist was done with two root filesystems, where testable kernel booted from 2. partition, and copy of the 2. partition was stored on 1. partition and could be restored at will.

I'll continue bisecting this bug on the linux-next tree again. If someone has more information where this issue originates from, help would be appreciated.

Regards,

Tomi Sarvela

--
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx