regression: CPU soft lockup with raid10: check slab-out-of-bounds in md_bitmap_get_counter

DISCUSSION Nigel Croxon 3 participants 10 messages Apr 19, 2024 – May 6, 2024 1 message not in the archive

Subsystems (2)

MD RAID Device mapper

lore

Share & tools

lore

Nigel Croxon

author

2024-04-19 19:49 UTC #0 lore raw

There is a problem with this commit, it causes a CPU#x soft lockup

commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter

Message from syslogd@rhel9 at Apr 19 14:14:55 ... kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [mdX_resync:6976]

dmesg:

[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc [ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX: 000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI: ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09: 0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12: 0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15: ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4: 0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240
[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140
[ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90
[ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0
[ 104.245668] raid10_sync_request+0x25c/0x1b40 [raid10]
[ 104.245676] ? is_mddev_idle+0x132/0x150
[ 104.245680] md_do_sync+0x64b/0x1020
[ 104.245683] ? __pfx_autoremove_wake_function+0x10/0x10
[ 104.245690] md_thread+0xa7/0x170
[ 104.245693] ? __pfx_md_thread+0x10/0x10
[ 104.245696] kthread+0xcf/0x100
[ 104.245700] ? __pfx_kthread+0x10/0x10
[ 104.245704] ret_from_fork+0x30/0x50
[ 104.245707] ? __pfx_kthread+0x10/0x10
[ 104.245710] ret_from_fork_asm+0x1a/0x30
[ 104.245714] </TASK>

When you run the reproducer script below...

#!/bin/sh
vg=t
lv=t
devs="/dev/sd[c-j]"
sz=3G
isz=2G
path=/dev/$vg/$lv
mnt=/mnt/$lv

vgcreate -y $vg $devs
lvcreate --yes --nosync --type raid10 -i 2 -n $lv -L $sz $vg

mkfs.xfs $path
mkdir -p $mnt
mount $path $mnt
df -h

for i in {1..10}
do
lvextend -y -L +$isz -r $path
lvs
done

lvs -a -o +devices
lvchange --syncaction check $path
#lvs -ovgname,lvname,copypercent t/t <-- this cmd to watch

Yu Kuai

2024-04-20 06:10 UTC #1 lore raw

Hi,

Quoted 7 lines · Nigel Croxon

There is a problem with this commit, it causes a CPU#x soft lockup
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter

Did you found this commit by bisect?

Quoted 27 lines

Message from syslogd@rhel9 at Apr 19 14:14:55 ...
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s!
[mdX_resync:6976]
dmesg:
[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc [ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX: 000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI: ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09: 0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12: 0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15: ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4: 0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240

Quoted 8 lines

[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140 [ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90 [ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0

Looks like you trigger the condition from md_bitmap_get_counter():

page >= bitmap->pages

by the command lvextend + lvchange --syncaction. And because md_bitmap_get_counter() return NULL with sync_blocks set to 0, raid10_sync_request() can't make progress and stuck in a dead loop.

There are two problems here:

1) Looks like lvextend doesn't resize the bitmap, I don't know about lvextend but this can explain why the condition can be triggered.

2) raid10_sync_request() should handle this case, by:
a) keeping syncing ranges beyond bitmap;
b) skip syncing reanges beyond bitmap;

Following is a patch to fix this problem by 2-b, which is the same before 301867b1c16805aebbc306aafa6ecdc68b73c7e5. However, 1) still need to be fixed, otherwise, data beyond bitmap ranges will never sync.

Nigel, can you give this patch a test?

Thanks,
Kuai

drivers/md/md-bitmap.c +0 -0

index 9672f75c3050..26e40991369a 100644
@@ -1424,15 +1424,17 @@ __acquires(bitmap->lock)
        sector_t chunk = offset >> bitmap->chunkshift;
        unsigned long page = chunk >> PAGE_COUNTER_SHIFT;
        unsigned long pageoff = (chunk & PAGE_COUNTER_MASK) <<

COUNTER_BYTE_SHIFT;
- sector_t csize;

+       sector_t csize = ((sector_t)1) << bitmap->chunkshift;
        int err;

+
if (page >= bitmap->pages) {
/*
* This can happen if bitmap_start_sync goes beyond
* End-of-device while looking for a whole page or
* user set a huge number to sysfs bitmap_set_bits. */

+               *blocks = csize - (offset & (csize - 1));
                return NULL;
        }
        err = md_bitmap_checkpage(bitmap, page, create, 0);
@@ -1441,8 +1443,7 @@ __acquires(bitmap->lock)
            bitmap->bp[page].map == NULL)
                csize = ((sector_t)1) << (bitmap->chunkshift +
                                          PAGE_COUNTER_SHIFT);
-       else
-               csize = ((sector_t)1) << bitmap->chunkshift;
+
        *blocks = csize - (offset & (csize - 1));

        if (err < 0)

Quoted 45 lines

[ 104.245668] raid10_sync_request+0x25c/0x1b40 [raid10] [ 104.245676] ? is_mddev_idle+0x132/0x150
[ 104.245680] md_do_sync+0x64b/0x1020
[ 104.245683] ? __pfx_autoremove_wake_function+0x10/0x10 [ 104.245690] md_thread+0xa7/0x170
[ 104.245693] ? __pfx_md_thread+0x10/0x10
[ 104.245696] kthread+0xcf/0x100
[ 104.245700] ? __pfx_kthread+0x10/0x10
[ 104.245704] ret_from_fork+0x30/0x50
[ 104.245707] ? __pfx_kthread+0x10/0x10
[ 104.245710] ret_from_fork_asm+0x1a/0x30
[ 104.245714] </TASK>
When you run the reproducer script below...
#!/bin/sh
vg=t
lv=t
devs="/dev/sd[c-j]"
sz=3G
isz=2G
path=/dev/$vg/$lv
mnt=/mnt/$lv
vgcreate -y $vg $devs
lvcreate --yes --nosync --type raid10 -i 2 -n $lv -L $sz $vg
mkfs.xfs $path
mkdir -p $mnt
mount $path $mnt
df -h
for i in {1..10}
do
   lvextend -y -L +$isz -r $path
   lvs
done
lvs -a -o +devices
lvchange --syncaction check $path
#lvs -ovgname,lvname,copypercent t/t         <-- this cmd to watch
.

Nigel Croxon

author

2024-04-21 12:30 UTC #2 lore raw

Quoted 5 lines · Yu Kuai

Hi,
Quoted 2 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup

Quoted 8 lines

Quoted 5 lines
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
Did you found this commit by bisect?

Yes, found this issue by bisecting...

Quoted 52 lines

Quoted 26 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ... kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [mdX_resync:6976]
dmesg:
[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc [ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX: 000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI: ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09: 0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12: 0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15: ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4: 0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240
Quoted 8 lines
[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140 [ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90 [ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0
Looks like you trigger the condition from md_bitmap_get_counter():
page >= bitmap->pages
by the command lvextend + lvchange --syncaction. And because md_bitmap_get_counter() return NULL with sync_blocks set to 0, raid10_sync_request() can't make progress and stuck in a dead loop.
There are two problems here:
1) Looks like lvextend doesn't resize the bitmap, I don't know about lvextend but this can explain why the condition can be triggered.
2) raid10_sync_request() should handle this case, by:
a) keeping syncing ranges beyond bitmap;
b) skip syncing reanges beyond bitmap;

Quoted 6 lines

Following is a patch to fix this problem by 2-b, which is the same before 301867b1c16805aebbc306aafa6ecdc68b73c7e5. However, 1) still need to be fixed, otherwise, data beyond bitmap ranges will never sync.
Nigel, can you give this patch a test?

Hello Kuai,

I tested your patch under this failing environment and it works.

-Nigel

Quoted 84 lines

Thanks,
Kuai
drivers/md/md-bitmap.c +3 -1
index 9672f75c3050..26e40991369a 100644
@@ -1424,15 +1424,17 @@ __acquires(bitmap->lock)
         sector_t chunk = offset >> bitmap->chunkshift;
         unsigned long page = chunk >> PAGE_COUNTER_SHIFT;
         unsigned long pageoff = (chunk & PAGE_COUNTER_MASK) << COUNTER_BYTE_SHIFT;
-       sector_t csize;
+       sector_t csize = ((sector_t)1) << bitmap->chunkshift;
         int err;
 
+
         if (page >= bitmap->pages) {
                 /*
                  * This can happen if bitmap_start_sync goes beyond
                  * End-of-device while looking for a whole page or
                  * user set a huge number to sysfs bitmap_set_bits.
                  */
+               *blocks = csize - (offset & (csize - 1));
                 return NULL;
         }
         err = md_bitmap_checkpage(bitmap, page, create, 0);
@@ -1441,8 +1443,7 @@ __acquires(bitmap->lock)
             bitmap->bp[page].map == NULL)
                csize = ((sector_t)1) << (bitmap->chunkshift + PAGE_COUNTER_SHIFT); -       else
-               csize = ((sector_t)1) << bitmap->chunkshift; +
        *blocks = csize - (offset & (csize - 1));
        if (err < 0)
Quoted 45 lines
[ 104.245668] raid10_sync_request+0x25c/0x1b40 [raid10] [ 104.245676] ? is_mddev_idle+0x132/0x150
[ 104.245680] md_do_sync+0x64b/0x1020
[ 104.245683] ? __pfx_autoremove_wake_function+0x10/0x10 [ 104.245690] md_thread+0xa7/0x170
[ 104.245693] ? __pfx_md_thread+0x10/0x10
[ 104.245696] kthread+0xcf/0x100
[ 104.245700] ? __pfx_kthread+0x10/0x10
[ 104.245704] ret_from_fork+0x30/0x50
[ 104.245707] ? __pfx_kthread+0x10/0x10
[ 104.245710] ret_from_fork_asm+0x1a/0x30
[ 104.245714] </TASK>
When you run the reproducer script below...
#!/bin/sh
vg=t
lv=t
devs="/dev/sd[c-j]"
sz=3G
isz=2G
path=/dev/$vg/$lv
mnt=/mnt/$lv
vgcreate -y $vg $devs
lvcreate --yes --nosync --type raid10 -i 2 -n $lv -L $sz $vg
mkfs.xfs $path
mkdir -p $mnt
mount $path $mnt
df -h
for i in {1..10}
do
    lvextend -y -L +$isz -r $path
    lvs
done
lvs -a -o +devices
lvchange --syncaction check $path
#lvs -ovgname,lvname,copypercent t/t         <-- this cmd to watch
.

Yu Kuai

2024-04-22 01:10 UTC #3 lore raw

Hi,

Quoted 4 lines · Nigel Croxon

Hello Kuai,
I tested your patch under this failing environment and it works.

That's greate, thanks for the test!

Kuai

Quoted 1 line

-Nigel

Yu Kuai

2024-04-24 06:57 UTC #4 lore raw

In reply to Nigel Croxon

Hi, Nigel

Quoted 13 lines · Nigel Croxon

Quoted 11 lines · Yu Kuai
Hi,
Quoted 7 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter

Quoted 38 lines

Quoted 2 lines
Did you found this commit by bisect?
Yes, found this issue by bisecting...
Quoted 34 lines
Quoted 34 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ... kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [mdX_resync:6976]
dmesg:
[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc [ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX: 000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI: ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09: 0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12: 0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15: ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4: 0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240
[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140
[ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90
[ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0

Can you give the following patch a test as well? I believe this is the root cause why page > bitmap->pages, dm-raid is using the wrong bitmap size.

drivers/md/dm-raid.c +0 -0

index abe88d1e6735..d9c65ef9c9fb 100644
@@ -4052,7 +4052,8 @@ static int raid_preresume(struct dm_target *ti)
               mddev->bitmap_info.chunksize !=

to_bytes(rs->requested_bitmap_chunk_sectors)))) {
int chunksize =
to_bytes(rs->requested_bitmap_chunk_sectors) ?: mddev->bitmap_info.chunksize;

- r = md_bitmap_resize(mddev->bitmap, mddev->dev_sectors, chunksize, 0);
+ r = md_bitmap_resize(mddev->bitmap, mddev->resync_max_sectors,

+                                    chunksize, 0);
                if (r)
                        DMERR("Failed to resize bitmap");
        }

Thanks,
Kuai

Quoted 117 lines

Quoted 18 lines
Looks like you trigger the condition from md_bitmap_get_counter():
page >= bitmap->pages
by the command lvextend + lvchange --syncaction. And because md_bitmap_get_counter() return NULL with sync_blocks set to 0, raid10_sync_request() can't make progress and stuck in a dead loop.
There are two problems here:
1) Looks like lvextend doesn't resize the bitmap, I don't know about lvextend but this can explain why the condition can be triggered.
2) raid10_sync_request() should handle this case, by:
a) keeping syncing ranges beyond bitmap;
b) skip syncing reanges beyond bitmap;
Quoted 6 lines
Following is a patch to fix this problem by 2-b, which is the same before 301867b1c16805aebbc306aafa6ecdc68b73c7e5. However, 1) still need to be fixed, otherwise, data beyond bitmap ranges will never sync.
Nigel, can you give this patch a test?
Hello Kuai,
I tested your patch under this failing environment and it works.
-Nigel
Quoted 35 lines
Thanks,
Kuai
drivers/md/md-bitmap.c +3 -1
index 9672f75c3050..26e40991369a 100644
@@ -1424,15 +1424,17 @@ __acquires(bitmap->lock)
         sector_t chunk = offset >> bitmap->chunkshift;
         unsigned long page = chunk >> PAGE_COUNTER_SHIFT;
         unsigned long pageoff = (chunk & PAGE_COUNTER_MASK) << COUNTER_BYTE_SHIFT;
-       sector_t csize;
+       sector_t csize = ((sector_t)1) << bitmap->chunkshift;
         int err;
 
+
         if (page >= bitmap->pages) {
                 /*
                  * This can happen if bitmap_start_sync goes beyond
                  * End-of-device while looking for a whole page or
                  * user set a huge number to sysfs bitmap_set_bits.
                  */
+               *blocks = csize - (offset & (csize - 1));
                 return NULL;
         }
         err = md_bitmap_checkpage(bitmap, page, create, 0);
@@ -1441,8 +1443,7 @@ __acquires(bitmap->lock)
             bitmap->bp[page].map == NULL)
                csize = ((sector_t)1) << (bitmap->chunkshift + PAGE_COUNTER_SHIFT); -       else
-               csize = ((sector_t)1) << bitmap->chunkshift; +
        *blocks = csize - (offset & (csize - 1));
Quoted 49 lines
        if (err < 0)
Quoted 45 lines
[ 104.245668] raid10_sync_request+0x25c/0x1b40 [raid10] [ 104.245676] ? is_mddev_idle+0x132/0x150
[ 104.245680] md_do_sync+0x64b/0x1020
[ 104.245683] ? __pfx_autoremove_wake_function+0x10/0x10 [ 104.245690] md_thread+0xa7/0x170
[ 104.245693] ? __pfx_md_thread+0x10/0x10
[ 104.245696] kthread+0xcf/0x100
[ 104.245700] ? __pfx_kthread+0x10/0x10
[ 104.245704] ret_from_fork+0x30/0x50
[ 104.245707] ? __pfx_kthread+0x10/0x10
[ 104.245710] ret_from_fork_asm+0x1a/0x30
[ 104.245714] </TASK>
When you run the reproducer script below...
#!/bin/sh
vg=t
lv=t
devs="/dev/sd[c-j]"
sz=3G
isz=2G
path=/dev/$vg/$lv
mnt=/mnt/$lv
vgcreate -y $vg $devs
lvcreate --yes --nosync --type raid10 -i 2 -n $lv -L $sz $vg
mkfs.xfs $path
mkdir -p $mnt
mount $path $mnt
df -h
for i in {1..10}
do
    lvextend -y -L +$isz -r $path
    lvs
done
lvs -a -o +devices
lvchange --syncaction check $path
#lvs -ovgname,lvname,copypercent t/t         <-- this cmd to watch
.
.

Nigel Croxon

author

2024-04-25 12:10 UTC #5 lore raw

Quoted 76 lines · Yu Kuai

Hi, Nigel
Quoted 13 lines · Nigel Croxon
Quoted 11 lines · Yu Kuai
Hi,
Quoted 7 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
Quoted 38 lines
Quoted 2 lines
Did you found this commit by bisect?
Yes, found this issue by bisecting...
Quoted 34 lines
Quoted 34 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ... kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [mdX_resync:6976]
dmesg:
[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc [ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX: 000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI: ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09: 0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12: 0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15: ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4: 0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240
[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140
[ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90
[ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0
Can you give the following patch a test as well? I believe this is the root cause why page > bitmap->pages, dm-raid is using the wrong bitmap size.
drivers/md/dm-raid.c +2 -1
index abe88d1e6735..d9c65ef9c9fb 100644
@@ -4052,7 +4052,8 @@ static int raid_preresume(struct dm_target *ti)
                mddev->bitmap_info.chunksize != to_bytes(rs->requested_bitmap_chunk_sectors)))) {
                 int chunksize = to_bytes(rs->requested_bitmap_chunk_sectors) ?: mddev->bitmap_info.chunksize;
 
-               r = md_bitmap_resize(mddev->bitmap, 
 mddev->dev_sectors, chunksize, 0);
+               r = md_bitmap_resize(mddev->bitmap, 
 mddev->resync_max_sectors,
+                                    chunksize, 0);
                 if (r)
DMERR("Failed to resize bitmap"); }

Quoted 2 lines

Thanks,
Kuai

Hello Kaui,

Tested and found no issues. Good to go..

-Nigel

Quoted 118 lines

Quoted 117 lines
Quoted 18 lines
Looks like you trigger the condition from md_bitmap_get_counter():
page >= bitmap->pages
by the command lvextend + lvchange --syncaction. And because md_bitmap_get_counter() return NULL with sync_blocks set to 0, raid10_sync_request() can't make progress and stuck in a dead loop.
There are two problems here:
1) Looks like lvextend doesn't resize the bitmap, I don't know about lvextend but this can explain why the condition can be triggered.
2) raid10_sync_request() should handle this case, by:
a) keeping syncing ranges beyond bitmap;
b) skip syncing reanges beyond bitmap;
Quoted 6 lines
Following is a patch to fix this problem by 2-b, which is the same before 301867b1c16805aebbc306aafa6ecdc68b73c7e5. However, 1) still need to be fixed, otherwise, data beyond bitmap ranges will never sync.
Nigel, can you give this patch a test?
Hello Kuai,
I tested your patch under this failing environment and it works.
-Nigel
Quoted 35 lines
Thanks,
Kuai
drivers/md/md-bitmap.c +3 -1
index 9672f75c3050..26e40991369a 100644
@@ -1424,15 +1424,17 @@ __acquires(bitmap->lock)
         sector_t chunk = offset >> bitmap->chunkshift;
         unsigned long page = chunk >> PAGE_COUNTER_SHIFT;
         unsigned long pageoff = (chunk & PAGE_COUNTER_MASK) << COUNTER_BYTE_SHIFT;
-       sector_t csize;
+       sector_t csize = ((sector_t)1) << bitmap->chunkshift;
         int err;
 
+
         if (page >= bitmap->pages) {
                 /*
                  * This can happen if bitmap_start_sync goes beyond
                  * End-of-device while looking for a whole page or
                  * user set a huge number to sysfs bitmap_set_bits.
                  */
+               *blocks = csize - (offset & (csize - 1));
                 return NULL;
         }
         err = md_bitmap_checkpage(bitmap, page, create, 0);
@@ -1441,8 +1443,7 @@ __acquires(bitmap->lock)
             bitmap->bp[page].map == NULL)
                csize = ((sector_t)1) << (bitmap->chunkshift + PAGE_COUNTER_SHIFT); -       else
-               csize = ((sector_t)1) << bitmap->chunkshift; +
        *blocks = csize - (offset & (csize - 1));
Quoted 49 lines
        if (err < 0)
Quoted 45 lines
[ 104.245668] raid10_sync_request+0x25c/0x1b40 [raid10] [ 104.245676] ? is_mddev_idle+0x132/0x150
[ 104.245680] md_do_sync+0x64b/0x1020
[ 104.245683] ? __pfx_autoremove_wake_function+0x10/0x10 [ 104.245690] md_thread+0xa7/0x170
[ 104.245693] ? __pfx_md_thread+0x10/0x10
[ 104.245696] kthread+0xcf/0x100
[ 104.245700] ? __pfx_kthread+0x10/0x10
[ 104.245704] ret_from_fork+0x30/0x50
[ 104.245707] ? __pfx_kthread+0x10/0x10
[ 104.245710] ret_from_fork_asm+0x1a/0x30
[ 104.245714] </TASK>
When you run the reproducer script below...
#!/bin/sh
vg=t
lv=t
devs="/dev/sd[c-j]"
sz=3G
isz=2G
path=/dev/$vg/$lv
mnt=/mnt/$lv
vgcreate -y $vg $devs
lvcreate --yes --nosync --type raid10 -i 2 -n $lv -L $sz $vg
mkfs.xfs $path
mkdir -p $mnt
mount $path $mnt
df -h
for i in {1..10}
do
    lvextend -y -L +$isz -r $path
    lvs
done
lvs -a -o +devices
lvchange --syncaction check $path
#lvs -ovgname,lvname,copypercent t/t         <-- this cmd to watch
.
.

Song Liu

maintainer

2024-04-25 16:52 UTC #6 lore raw

Quoted 97 lines · Nigel Croxon

Quoted 94 lines · Yu Kuai
Hi, Nigel
Quoted 13 lines · Nigel Croxon
Quoted 11 lines · Yu Kuai
Hi,
Quoted 7 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
Quoted 53 lines
Quoted 2 lines
Did you found this commit by bisect?
Yes, found this issue by bisecting...
Quoted 49 lines
Quoted 49 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ...
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s!
[mdX_resync:6976]
dmesg:
[  104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not
tainted 6.9.0-rc4-next-20240419 #1
[  104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS 1.16.2-1.fc38 04/01/2014
[  104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[  104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90
90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff
0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc
cc cc cc cc cc
[  104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[  104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX:
000000000000000c
[  104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI:
ffff926160ccd200
[  104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09:
0000000000000000
[  104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12:
0000000000000000
[  104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15:
ffff926151019000
[  104.245611] FS:  0000000000000000(0000)
GS:ffff9273f9580000(0000) knlGS:0000000000000000
[  104.245613] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4:
0000000000370ef0
[  104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  104.245618] Call Trace:
[  104.245620]  <IRQ>
[  104.245623]  ? watchdog_timer_fn+0x1e3/0x260
[  104.245630]  ? __pfx_watchdog_timer_fn+0x10/0x10
[  104.245634]  ? __hrtimer_run_queues+0x112/0x2a0
[  104.245638]  ? hrtimer_interrupt+0xff/0x240
[  104.245640]  ? sched_clock+0xc/0x30
[  104.245644]  ? __sysvec_apic_timer_interrupt+0x54/0x140
[  104.245649]  ? sysvec_apic_timer_interrupt+0x6c/0x90
[  104.245652]  </IRQ>
[  104.245653]  <TASK>
[  104.245654]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[  104.245659]  ? _raw_spin_unlock_irq+0x13/0x30
[  104.245661]  md_bitmap_start_sync+0x6b/0xf0
Can you give the following patch a test as well? I believe this is the root cause why page > bitmap->pages, dm-raid is using the wrong bitmap size.
drivers/md/dm-raid.c +0 -0
index abe88d1e6735..d9c65ef9c9fb 100644
@@ -4052,7 +4052,8 @@ static int raid_preresume(struct dm_target *ti)
               mddev->bitmap_info.chunksize !=
 to_bytes(rs->requested_bitmap_chunk_sectors)))) {
                int chunksize =
to_bytes(rs->requested_bitmap_chunk_sectors) ?:
mddev->bitmap_info.chunksize;
- r = md_bitmap_resize(mddev->bitmap,
mddev->dev_sectors, chunksize, 0);
+ r = md_bitmap_resize(mddev->bitmap,
mddev->resync_max_sectors,
+                                    chunksize, 0);
                if (r)
                        DMERR("Failed to resize bitmap");
        }

Quoted 8 lines

Quoted 2 lines
Thanks,
Kuai
Hello Kaui,
Tested and found no issues. Good to go..
-Nigel

Thanks for the fixes and the tests.

For the next step, do we need both patches or just one of them?

Song

Nigel Croxon

author

2024-04-30 11:07 UTC #7 lore raw

Quoted 98 lines · Song Liu

Quoted 97 lines · Nigel Croxon
Quoted 86 lines · Yu Kuai
Hi, Nigel
Quoted 6 lines · Nigel Croxon
Quoted 5 lines · Yu Kuai
Hi,
Quoted 2 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup
Quoted 58 lines
Quoted 7 lines
Quoted 5 lines
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
Did you found this commit by bisect?
Yes, found this issue by bisecting...
Quoted 49 lines
Quoted 49 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ...
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s!
[mdX_resync:6976]
dmesg:
[  104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not
tainted 6.9.0-rc4-next-20240419 #1
[  104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS 1.16.2-1.fc38 04/01/2014
[  104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
[  104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90
90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff
0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc
cc cc cc cc cc
[  104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
[  104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX:
000000000000000c
[  104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI:
ffff926160ccd200
[  104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09:
0000000000000000
[  104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12:
0000000000000000
[  104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15:
ffff926151019000
[  104.245611] FS:  0000000000000000(0000)
GS:ffff9273f9580000(0000) knlGS:0000000000000000
[  104.245613] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4:
0000000000370ef0
[  104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  104.245618] Call Trace:
[  104.245620]  <IRQ>
[  104.245623]  ? watchdog_timer_fn+0x1e3/0x260
[  104.245630]  ? __pfx_watchdog_timer_fn+0x10/0x10
[  104.245634]  ? __hrtimer_run_queues+0x112/0x2a0
[  104.245638]  ? hrtimer_interrupt+0xff/0x240
[  104.245640]  ? sched_clock+0xc/0x30
[  104.245644]  ? __sysvec_apic_timer_interrupt+0x54/0x140
[  104.245649]  ? sysvec_apic_timer_interrupt+0x6c/0x90
[  104.245652]  </IRQ>
[  104.245653]  <TASK>
[  104.245654]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[  104.245659]  ? _raw_spin_unlock_irq+0x13/0x30
[  104.245661]  md_bitmap_start_sync+0x6b/0xf0
Can you give the following patch a test as well? I believe this is the root cause why page > bitmap->pages, dm-raid is using the wrong bitmap size.
drivers/md/dm-raid.c +0 -0
index abe88d1e6735..d9c65ef9c9fb 100644
@@ -4052,7 +4052,8 @@ static int raid_preresume(struct dm_target *ti)
                mddev->bitmap_info.chunksize !=
 to_bytes(rs->requested_bitmap_chunk_sectors)))) {
                 int chunksize =
to_bytes(rs->requested_bitmap_chunk_sectors) ?:
mddev->bitmap_info.chunksize;
- r = md_bitmap_resize(mddev->bitmap,
mddev->dev_sectors, chunksize, 0);
+ r = md_bitmap_resize(mddev->bitmap,
mddev->resync_max_sectors,
Quoted 7 lines
+                                    chunksize, 0);
                 if (r)
                         DMERR("Failed to resize bitmap");
         }
Thanks,
Kuai
Hello Kaui,

Quoted 9 lines

Quoted 3 lines
Tested and found no issues. Good to go..
-Nigel
Thanks for the fixes and the tests.
For the next step, do we need both patches or just one of them?
Song

They both fix the problem independently without the other.

-Nigel

Yu Kuai

2024-05-06 06:19 UTC #8 lore raw

Hi,

Quoted 102 lines · Nigel Croxon

Quoted 100 lines · Song Liu
Quoted 99 lines · Nigel Croxon
Quoted 91 lines · Yu Kuai
Hi, Nigel
Quoted 6 lines · Nigel Croxon
Quoted 5 lines · Yu Kuai
Hi,
Quoted 2 lines · Nigel Croxon
There is a problem with this commit, it causes a CPU#x soft lockup
Quoted 58 lines
Quoted 7 lines
Quoted 5 lines
commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
Author: Li Nan <[email protected]>
Date: Mon May 15 21:48:05 2023 +0800
md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
Did you found this commit by bisect?
Yes, found this issue by bisecting...
Quoted 49 lines
Quoted 49 lines
Message from syslogd@rhel9 at Apr 19 14:14:55 ... kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [mdX_resync:6976]
dmesg:
[ 104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump: loaded Not tainted 6.9.0-rc4-next-20240419 #1
[ 104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[ 104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30 [ 104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb 65 ff 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc cc cc cc cc cc
[ 104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246 [ 104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX:
000000000000000c
[ 104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI:
ffff926160ccd200
[ 104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09:
0000000000000000
[ 104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12:
0000000000000000
[ 104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15:
ffff926151019000
[ 104.245611] FS: 0000000000000000(0000) GS:ffff9273f9580000(0000) knlGS:0000000000000000
[ 104.245613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4:
0000000000370ef0
[ 104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 104.245618] Call Trace:
[ 104.245620] <IRQ>
[ 104.245623] ? watchdog_timer_fn+0x1e3/0x260
[ 104.245630] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 104.245634] ? __hrtimer_run_queues+0x112/0x2a0
[ 104.245638] ? hrtimer_interrupt+0xff/0x240
[ 104.245640] ? sched_clock+0xc/0x30
[ 104.245644] ? __sysvec_apic_timer_interrupt+0x54/0x140 [ 104.245649] ? sysvec_apic_timer_interrupt+0x6c/0x90 [ 104.245652] </IRQ>
[ 104.245653] <TASK>
[ 104.245654] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 104.245659] ? _raw_spin_unlock_irq+0x13/0x30
[ 104.245661] md_bitmap_start_sync+0x6b/0xf0
Can you give the following patch a test as well? I believe this is the root cause why page > bitmap->pages, dm-raid is using the wrong bitmap size.
drivers/md/dm-raid.c +0 -0
index abe88d1e6735..d9c65ef9c9fb 100644
@@ -4052,7 +4052,8 @@ static int raid_preresume(struct dm_target *ti)
                mddev->bitmap_info.chunksize !=
to_bytes(rs->requested_bitmap_chunk_sectors)))) {
                 int chunksize =
to_bytes(rs->requested_bitmap_chunk_sectors) ?:
mddev->bitmap_info.chunksize;
-               r = md_bitmap_resize(mddev->bitmap, mddev->dev_sectors, chunksize, 0);
+               r = md_bitmap_resize(mddev->bitmap, mddev->resync_max_sectors,
+                                    chunksize, 0); if (r)
                         DMERR("Failed to resize bitmap"); }
Quoted 2 lines
Thanks,
Kuai
Hello Kaui,
Tested and found no issues. Good to go..

Quoted 8 lines

Quoted 7 lines
Quoted 1 line
-Nigel
Thanks for the fixes and the tests.
For the next step, do we need both patches or just one of them?
Song
They both fix the problem independently without the other.

Sorry that I forgot to reply here, we discussed this on slack...

For md/raid, we already apply the first patch to fix the soft lockup problem, for dm-raid, other than the second patch to fix wrong bitmap size, we still need more changes, because some fields in mddev for dm-raid10 and dm-raid5 are different, while dm-raid doesn't distinguish them. I'm working on that, however, I'm not that familiar with dm-raid and I need more time. :)

Thanks,
Kuai

Quoted 5 lines

-Nigel
.

A message that is not in the archive (sent off-list or not yet received): CAM23Vxr6unOdZoBQZBML_uxDGc_pquktfY=R6n=kDPcWG0Jsvg@mail.gmail.com

Yu Kuai

2024-05-06 13:52 UTC #10 lore raw

Hi,

Quoted 3 lines · Heinz Mauelshagen

Hi,
what fields are you referring to?

For this problem, the field is: dev_sectors, for raid10, it's rdev size, for raid456, it's array size. And dm-raid is using it as bitmap size.

And while review related code, I found following quite strage as well:

mddev->resync_max_sectors = mddev->dev_sectors;

I'm still checking following fields now, for both md/raid and dm-raid, with the respect how sync_thread should work:

dev_sectors
resync_max_sectors
array_sectors
recovery_cp
recovery_offset
reshape_position

Thanks,
Kuai

Quoted 149 lines

Thanks,
Heinz

On Mon, May 6, 2024 at 8:19 AM Yu Kuai <[email protected] <mailto:[email protected]>> wrote:

    Hi,

    在 2024/04/30 19:07, Nigel Croxon 写道:
     >
     > On 4/25/24 12:52 PM, Song Liu wrote:
     >> On Thu, Apr 25, 2024 at 5:10 AM Nigel Croxon <[email protected]
    <mailto:[email protected]>> wrote:
     >>>
     >>> On 4/24/24 2:57 AM, Yu Kuai wrote:
     >>>> Hi, Nigel
     >>>>
     >>>> 在 2024/04/21 20:30, Nigel Croxon 写道:
     >>>>> On 4/20/24 2:09 AM, Yu Kuai wrote:
     >>>>>> Hi,
     >>>>>>
     >>>>>> 在 2024/04/20 3:49, Nigel Croxon 写道:
     >>>>>>> There is a problem with this commit, it causes a CPU#x soft
    lockup
     >>>>>>>
     >>>>>>> commit 301867b1c16805aebbc306aafa6ecdc68b73c7e5
     >>>>>>> Author: Li Nan <[email protected]
    <mailto:[email protected]>>
     >>>>>>> Date:   Mon May 15 21:48:05 2023 +0800
     >>>>>>> md/raid10: check slab-out-of-bounds in md_bitmap_get_counter
     >>>>>>>
     >>>>>> Did you found this commit by bisect?
     >>>>>>
     >>>>> Yes, found this issue by bisecting...
     >>>>>
     >>>>>>> Message from syslogd@rhel9 at Apr 19 14:14:55 ...
     >>>>>>>    kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 26s!
     >>>>>>> [mdX_resync:6976]
     >>>>>>>
     >>>>>>> dmesg:
     >>>>>>>
     >>>>>>> [  104.245585] CPU: 7 PID: 3588 Comm: mdX_resync Kdump:
    loaded Not
     >>>>>>> tainted 6.9.0-rc4-next-20240419 #1
     >>>>>>> [  104.245588] Hardware name: QEMU Standard PC (Q35 + ICH9,
    2009),
     >>>>>>> BIOS 1.16.2-1.fc38 04/01/2014
     >>>>>>> [  104.245590] RIP: 0010:_raw_spin_unlock_irq+0x13/0x30
     >>>>>>> [  104.245598] Code: 00 00 00 00 00 66 90 90 90 90 90 90 90
    90 90
     >>>>>>> 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 90 90 90 fb
    65 ff
     >>>>>>> 0d 95 9f 75 76 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc
    cc cc cc
     >>>>>>> cc cc cc cc cc
     >>>>>>> [  104.245601] RSP: 0018:ffffb2d74a81bbf8 EFLAGS: 00000246
     >>>>>>> [  104.245603] RAX: 0000000000000000 RBX: 0000000001000000 RCX:
     >>>>>>> 000000000000000c
     >>>>>>> [  104.245604] RDX: 0000000000000000 RSI: 0000000001000000 RDI:
     >>>>>>> ffff926160ccd200
     >>>>>>> [  104.245606] RBP: ffffb2d74a81bcd0 R08: 0000000000000013 R09:
     >>>>>>> 0000000000000000
     >>>>>>> [  104.245607] R10: 0000000000000000 R11: ffffb2d74a81bad8 R12:
     >>>>>>> 0000000000000000
     >>>>>>> [  104.245608] R13: 0000000000000000 R14: ffff926160ccd200 R15:
     >>>>>>> ffff926151019000
     >>>>>>> [  104.245611] FS:  0000000000000000(0000)
     >>>>>>> GS:ffff9273f9580000(0000) knlGS:0000000000000000
     >>>>>>> [  104.245613] CS:  0010 DS: 0000 ES: 0000 CR0:
    0000000080050033
     >>>>>>> [  104.245614] CR2: 00007f23774d2584 CR3: 0000000104098003 CR4:
     >>>>>>> 0000000000370ef0
     >>>>>>> [  104.245616] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
     >>>>>>> 0000000000000000
     >>>>>>> [  104.245617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
     >>>>>>> 0000000000000400
     >>>>>>> [  104.245618] Call Trace:
     >>>>>>> [  104.245620]  <IRQ>
     >>>>>>> [  104.245623]  ? watchdog_timer_fn+0x1e3/0x260
     >>>>>>> [  104.245630]  ? __pfx_watchdog_timer_fn+0x10/0x10
     >>>>>>> [  104.245634]  ? __hrtimer_run_queues+0x112/0x2a0
     >>>>>>> [  104.245638]  ? hrtimer_interrupt+0xff/0x240
     >>>>>>> [  104.245640]  ? sched_clock+0xc/0x30
     >>>>>>> [  104.245644]  ? __sysvec_apic_timer_interrupt+0x54/0x140
     >>>>>>> [  104.245649]  ? sysvec_apic_timer_interrupt+0x6c/0x90
     >>>>>>> [  104.245652]  </IRQ>
     >>>>>>> [  104.245653]  <TASK>
     >>>>>>> [  104.245654]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
     >>>>>>> [  104.245659]  ? _raw_spin_unlock_irq+0x13/0x30
     >>>>>>> [  104.245661]  md_bitmap_start_sync+0x6b/0xf0
     >>>> Can you give the following patch a test as well? I believe this is
     >>>> the root cause why page > bitmap->pages, dm-raid is using the
    wrong
     >>>> bitmap size.
     >>>>
     >>>> diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
     >>>> index abe88d1e6735..d9c65ef9c9fb 100644
     >>>> --- a/drivers/md/dm-raid.c
     >>>> +++ b/drivers/md/dm-raid.c
     >>>> @@ -4052,7 +4052,8 @@ static int raid_preresume(struct
    dm_target *ti)
     >>>>                 mddev->bitmap_info.chunksize !=
     >>>> to_bytes(rs->requested_bitmap_chunk_sectors)))) {
     >>>>                  int chunksize =
     >>>> to_bytes(rs->requested_bitmap_chunk_sectors) ?:
     >>>> mddev->bitmap_info.chunksize;
     >>>>
     >>>> -               r = md_bitmap_resize(mddev->bitmap,
     >>>> mddev->dev_sectors, chunksize, 0);
     >>>> +               r = md_bitmap_resize(mddev->bitmap,
     >>>> mddev->resync_max_sectors,
     >>>> +                                    chunksize, 0);
     >>>>                  if (r)
     >>>>                          DMERR("Failed to resize bitmap");
     >>>>          }
     >>>>
     >>>> Thanks,
     >>>> Kuai
     >>> Hello Kaui,
     >>>
     >>> Tested and found no issues. Good to go..
     >>>
     >>> -Nigel
     >> Thanks for the fixes and the tests.
     >>
     >> For the next step, do we need both patches or just one of them?
     >>
     >> Song
     >>
     > They both fix the problem independently without the other.

    Sorry that I forgot to reply here, we discussed this on slack...

    For md/raid, we already apply the first patch to fix the soft lockup
    problem, for dm-raid, other than the second patch to fix wrong bitmap
    size, we still need more changes, because some fields in mddev for
    dm-raid10 and dm-raid5 are different, while dm-raid doesn't distinguish
    them. I'm working on that, however, I'm not that familiar with dm-raid
    and I need more time. :)

    Thanks,
    Kuai

     >
     > -Nigel
     >
     > .
     >