Skip to contentSkip to conversation

blktest failures

DISCUSSION Bob Pearson 5 participants 15 messages Apr 8, 2022 – Apr 15, 2022 InfiniBand

lore
Share & tools
lore
#0 lore raw

Bart,

I finally was able to build a kernel with lockdep enabled correctly and saw the error that you and others reported. I am not familiar with lockdep output but I am guessing that it is reporting a mismatch between a _bh spinlock and a _irqsave spinlock (since those are the only two types used by the driver.)

I went on campaign a while back to replace all the locks with _bh locks because I figured they would be faster than _irqsave locks and because the driver never touched a lock except from a verbs API call or from a tasklet (softirq.) As it turned out some code makes verbs API calls while in hardirq context which broke that assumption. So some of the locks were reverted back to irqsave locks which fixed those warnings.

Now it is happening again. I did an experiment and went through the rxe driver and replaced all spinlocks with _irqsave locks. Now the lockdep splats have gone away and the srp/001 test reports success. BUT, it hangs and doesn't finish. If I try to run all the tests I get warnings about unable to remove the scsi_debug driver. I am able to remove the rdma_rxe driver and reload it. I am not seeing any errors in the rxe driver.

Do you have any ideas what to look at next?

Bob

#1 lore raw
Quoted 17 lines · Bob Pearson

Bart,

I finally was able to build a kernel with lockdep enabled correctly and saw the error that you and others reported. I am not familiar with lockdep output but I am guessing that it is reporting a mismatch between a _bh spinlock and a _irqsave spinlock (since those are the only two types used by the driver.)

I went on campaign a while back to replace all the locks with _bh locks because I figured they would be faster than _irqsave locks and because the driver never touched a lock except from a verbs API call or from a tasklet (softirq.) As it turned out some code makes verbs API calls while in hardirq context which broke that assumption. So some of the locks were reverted back to irqsave locks which fixed those warnings.

Now it is happening again. I did an experiment and went through the rxe driver and replaced all spinlocks with _irqsave locks. Now the lockdep splats have gone away and the srp/001 test reports success. BUT, it hangs and doesn't finish. If I try to run all the tests I get warnings about unable to remove the scsi_debug driver. I am able to remove the rdma_rxe driver and reload it. I am not seeing any errors in the rxe driver.

Quoted 3 lines

Do you have any ideas what to look at next?

Bob

Actually it doesn't hang forever but I get the following

......

[  107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache
[  291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[  292.247547] rdma_rxe: unloaded

So it waits for about 3 minutes for something and then gives up.

Bob

#2 lore raw
Quoted 10 lines · Bob Pearson

Actually it doesn't hang forever but I get the following

......

[  107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache
[  291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[  292.247547] rdma_rxe: unloaded

So it waits for about 3 minutes for something and then gives up.

(+Christoph)

Hi Bob,

I can reproduce this behavior with the Soft-iWARP driver by running the following command:

cd blktests && use_siw=1 ./check -q srp/001

Christoph, the call stack involved in this issue is as follows:

__schedule+0x4c3/0xd20
schedule+0x82/0x110
schedule_timeout+0x122/0x200
io_schedule_timeout+0x7b/0xc0
__wait_for_common+0x2bc/0x380
wait_for_completion_io_timeout+0x1d/0x20
blk_execute_rq+0x1db/0x200
__scsi_execute+0x1fb/0x310
sd_sync_cache+0x155/0x2c0 [sd_mod]
sd_shutdown+0xbb/0x190 [sd_mod]
sd_remove+0x5b/0x80 [sd_mod]
device_remove+0x9a/0xb0
device_release_driver_internal+0x2c5/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1aa/0x270
device_del+0x2d4/0x640
__scsi_remove_device+0x168/0x1a0
scsi_forget_host+0xa8/0xb0
scsi_remove_host+0x9b/0x150
sdebug_driver_remove+0x3d/0x140 [scsi_debug]
device_remove+0x6f/0xb0
device_release_driver_internal+0x2c5/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1aa/0x270
device_del+0x2d4/0x640
device_unregister+0x18/0x70
sdebug_do_remove_host+0x138/0x180 [scsi_debug]
scsi_debug_exit+0x45/0xd5 [scsi_debug]
__do_sys_delete_module.constprop.0+0x210/0x320
__x64_sys_delete_module+0x1f/0x30
do_syscall_64+0x35/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Thanks,

Bart.

#3 lore raw
Quoted 65 lines · Bart Van Assche
Quoted 10 lines · Bob Pearson

Actually it doesn't hang forever but I get the following

......

[  107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache
[  291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[  292.247547] rdma_rxe: unloaded

So it waits for about 3 minutes for something and then gives up.

(+Christoph)

Hi Bob,

I can reproduce this behavior with the Soft-iWARP driver by running the following command:

cd blktests && use_siw=1 ./check -q srp/001

Christoph, the call stack involved in this issue is as follows:

__schedule+0x4c3/0xd20
schedule+0x82/0x110
schedule_timeout+0x122/0x200
io_schedule_timeout+0x7b/0xc0
__wait_for_common+0x2bc/0x380
wait_for_completion_io_timeout+0x1d/0x20
blk_execute_rq+0x1db/0x200
__scsi_execute+0x1fb/0x310
sd_sync_cache+0x155/0x2c0 [sd_mod]
sd_shutdown+0xbb/0x190 [sd_mod]
sd_remove+0x5b/0x80 [sd_mod]
device_remove+0x9a/0xb0
device_release_driver_internal+0x2c5/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1aa/0x270
device_del+0x2d4/0x640
__scsi_remove_device+0x168/0x1a0
scsi_forget_host+0xa8/0xb0
scsi_remove_host+0x9b/0x150
sdebug_driver_remove+0x3d/0x140 [scsi_debug]
device_remove+0x6f/0xb0
device_release_driver_internal+0x2c5/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1aa/0x270
device_del+0x2d4/0x640
device_unregister+0x18/0x70
sdebug_do_remove_host+0x138/0x180 [scsi_debug]
scsi_debug_exit+0x45/0xd5 [scsi_debug]
__do_sys_delete_module.constprop.0+0x210/0x320
__x64_sys_delete_module+0x1f/0x30
do_syscall_64+0x35/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Quoted 4 lines

Thanks,

Bart.

I reported/bisected for this issue last week, not sure if it helped.
https://lore.kernel.org/linux-block/CAHj4cs9OTm9sb_5fmzgz+W9OSLeVPKix3Yri856kqQVccwd_Mw@mail.gmail.com/T/#t

Signature · 2 lines
Best Regards,
  Yi Zhang
#5 lore raw
In reply to Bart Van Assche
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

#6 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

#7 lore raw
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Quoted 8 lines
Quoted 2 lines

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Hi Bob,

After having taken a closer look at del_gendisk(), I agree with what Christoph wrote above. Please revert patch "scsi: scsi_debug: Address races following module load" locally when running blktests. See also https://lore.kernel.org/linux-scsi/[email protected]/T/#m47a23ffd5ce68b8183100444d6e711b6b4aba393.

Thanks,

Bart.

#8 lore raw
In reply to Bob Pearson
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Quoted 8 lines
Quoted 2 lines

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

#9 lore raw
Quoted 20 lines · Yanjun Zhu
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Quoted 8 lines
Quoted 2 lines

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I know. I am trying to find out why. For performance reasons I very much want to make the xarray + rcu_locking patches work correctly. All the spinlock issues are now fixed in my tree. What is left is a race in the RDMA read retry code somewhere. I'll find it. In the process of chasing this I have found several real bugs and I suspect a few more are out there.

Bob

#10 lore raw
In reply to Yanjun Zhu
Quoted 20 lines · Yanjun Zhu
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

Quoted 8 lines
Quoted 2 lines

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.

#11 lore raw
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 6 lines
Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.

I am not sure about this because wr NULL problem exists with xarray patches.

Please let us find the root cause of wr NULL.

This can make RXE more stable.

Zhu Yanjun

#12 lore raw
Quoted 27 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 6 lines
Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.

Quoted 8 lines

I am not sure about this because wr NULL problem exists with xarray patches.

Please let us find the root cause of wr NULL.

This can make RXE more stable.

Zhu Yanjun

You mean mr = NULL. And it is not happening in my tree. I have WARN_ONs looking for it and it isn't happening.

#13 lore raw
Quoted 30 lines · Bob Pearson
Quoted 29 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 6 lines
Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.

I am not sure about this because wr NULL problem exists with xarray patches.

Quoted 8 lines
Quoted 6 lines

Please let us find the root cause of wr NULL.

This can make RXE more stable.

Zhu Yanjun

You mean mr = NULL. And it is not happening in my tree. I have WARN_ONs looking for it

Why you said that you can not reproduce this problem now?

Please check your mail. I remember that you can reproduce this wr NULL in your host.

Please focus on this problem. This can make RXE more stable. Let us find the root cause and fix this problem ASAP.

Thanks.
Zhu Yanjun

Quoted 1 line

and it isn't happening.

#14 lore raw
Quoted 35 lines · Yanjun Zhu
Quoted 34 lines · Bob Pearson
Quoted 33 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche

One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?

That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.

When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??

Quoted 6 lines
Quoted 4 lines

If you remove all the xarray patches and use the original source code. This will not occur.

Zhu Yanjun

I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.

I am not sure about this because wr NULL problem exists with xarray patches.

Please let us find the root cause of wr NULL.

This can make RXE more stable.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Zhu Yanjun

You mean mr = NULL. And it is not happening in my tree. I have WARN_ONs looking for it

Why you said that you can not reproduce this problem now?

Please check your mail. I remember that you can reproduce this wr NULL in your host.

Yes I did see it before but not with the new pool patches. The whole point of the patch series I have been working on for the past month was to clean up races in shutdown and cleanup code. These were demonstrated by colleagues here at hpe who are running lustre over soft roce. I looked at the shutdown code and saw that the whole design was flawed. We were using kref's to track reference counts on MRs and QPs etc. but always returning to rdma-core regardless of the reference count. rdma-core having no visibility to our references then deletes the object fairly soon after that while we are still trying to process late arriving packets. One of the features in my tree is to use wait_for_completion() in the return path to rdma-core and pause until all the references have been dropped.

I really don't see the point of debugging the old code because it is just wrong wrong wrong. It will never be stable under heavy load.

Bob

Quoted 9 lines

Please focus on this problem. This can make RXE more stable. Let us find the root cause and fix this problem ASAP.

Thanks.
Zhu Yanjun

Quoted 1 line

and it isn't happening.