blktest failures
Bart,
I finally was able to build a kernel with lockdep enabled correctly and saw the error that you and others reported. I am not familiar with lockdep output but I am guessing that it is reporting a mismatch between a _bh spinlock and a _irqsave spinlock (since those are the only two types used by the driver.)
I went on campaign a while back to replace all the locks with _bh locks because I figured they would be faster than _irqsave locks and because the driver never touched a lock except from a verbs API call or from a tasklet (softirq.) As it turned out some code makes verbs API calls while in hardirq context which broke that assumption. So some of the locks were reverted back to irqsave locks which fixed those warnings.
Now it is happening again. I did an experiment and went through the rxe driver and replaced all spinlocks with _irqsave locks. Now the lockdep splats have gone away and the srp/001 test reports success. BUT, it hangs and doesn't finish. If I try to run all the tests I get warnings about unable to remove the scsi_debug driver. I am able to remove the rdma_rxe driver and reload it. I am not seeing any errors in the rxe driver.
Do you have any ideas what to look at next?
Bob
Quoted 17 lines · Bob Pearson
Bart,
I finally was able to build a kernel with lockdep enabled correctly and saw the error that you and others reported. I am not familiar with lockdep output but I am guessing that it is reporting a mismatch between a
_bhspinlock and a_irqsavespinlock (since those are the only two types used by the driver.)I went on campaign a while back to replace all the locks with
_bhlocks because I figured they would be faster than_irqsavelocks and because the driver never touched a lock except from a verbs API call or from a tasklet (softirq.) As it turned out some code makes verbs API calls while in hardirq context which broke that assumption. So some of the locks were reverted back to irqsave locks which fixed those warnings.Now it is happening again. I did an experiment and went through the rxe driver and replaced all spinlocks with
_irqsavelocks. Now the lockdep splats have gone away and the srp/001 test reports success. BUT, it hangs and doesn't finish. If I try to run all the tests I get warnings about unable to remove thescsi_debugdriver. I am able to remove therdma_rxedriver and reload it. I am not seeing any errors in the rxe driver.
Quoted 3 lines
Do you have any ideas what to look at next?
Bob
Actually it doesn't hang forever but I get the following
......
[ 107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache
[ 291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[ 292.247547] rdma_rxe: unloaded
So it waits for about 3 minutes for something and then gives up.
Bob
Quoted 10 lines · Bob Pearson
Actually it doesn't hang forever but I get the following
......
[ 107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache[ 291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK[ 292.247547] rdma_rxe: unloadedSo it waits for about 3 minutes for something and then gives up.
(+Christoph)
Hi Bob,
I can reproduce this behavior with the Soft-iWARP driver by running the following command:
cd blktests && use_siw=1 ./check -q srp/001
Christoph, the call stack involved in this issue is as follows:
__schedule+0x4c3/0xd20 schedule+0x82/0x110 schedule_timeout+0x122/0x200 io_schedule_timeout+0x7b/0xc0 __wait_for_common+0x2bc/0x380 wait_for_completion_io_timeout+0x1d/0x20 blk_execute_rq+0x1db/0x200 __scsi_execute+0x1fb/0x310 sd_sync_cache+0x155/0x2c0 [sd_mod] sd_shutdown+0xbb/0x190 [sd_mod] sd_remove+0x5b/0x80 [sd_mod] device_remove+0x9a/0xb0 device_release_driver_internal+0x2c5/0x360 device_release_driver+0x12/0x20 bus_remove_device+0x1aa/0x270 device_del+0x2d4/0x640 __scsi_remove_device+0x168/0x1a0 scsi_forget_host+0xa8/0xb0 scsi_remove_host+0x9b/0x150 sdebug_driver_remove+0x3d/0x140 [scsi_debug] device_remove+0x6f/0xb0 device_release_driver_internal+0x2c5/0x360 device_release_driver+0x12/0x20 bus_remove_device+0x1aa/0x270 device_del+0x2d4/0x640 device_unregister+0x18/0x70 sdebug_do_remove_host+0x138/0x180 [scsi_debug] scsi_debug_exit+0x45/0xd5 [scsi_debug] __do_sys_delete_module.constprop.0+0x210/0x320 __x64_sys_delete_module+0x1f/0x30 do_syscall_64+0x35/0x80 entry_SYSCALL_64_after_hwframe+0x44/0xae
One of the functions in the above call stack is sd_remove(). sd_remove() calls del_gendisk() just before calling sd_shutdown(). sd_shutdown() submits the SYNCHRONIZE CACHE command. In del_gendisk() I found the following comment: "Fail any new I/O". Do you agree that failing new I/O before sd_shutdown() is called is wrong? Is there any other way to fix this than moving the blk_queue_start_drain() etc. calls out of del_gendisk() and into a new function?
Thanks,
Bart.
Quoted 65 lines · Bart Van Assche
Quoted 10 lines · Bob Pearson
Actually it doesn't hang forever but I get the following
......
[ 107.579576] sd 4:0:0:0: [sdb] Synchronizing SCSI cache[ 291.970133] sd 4:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK[ 292.247547] rdma_rxe: unloadedSo it waits for about 3 minutes for something and then gives up.
(+Christoph)
Hi Bob,
I can reproduce this behavior with the Soft-iWARP driver by running the following command:
cd blktests && use_siw=1 ./check -q srp/001
Christoph, the call stack involved in this issue is as follows:
__schedule+0x4c3/0xd20 schedule+0x82/0x110 schedule_timeout+0x122/0x200 io_schedule_timeout+0x7b/0xc0 __wait_for_common+0x2bc/0x380 wait_for_completion_io_timeout+0x1d/0x20 blk_execute_rq+0x1db/0x200 __scsi_execute+0x1fb/0x310 sd_sync_cache+0x155/0x2c0 [sd_mod] sd_shutdown+0xbb/0x190 [sd_mod] sd_remove+0x5b/0x80 [sd_mod] device_remove+0x9a/0xb0 device_release_driver_internal+0x2c5/0x360 device_release_driver+0x12/0x20 bus_remove_device+0x1aa/0x270 device_del+0x2d4/0x640 __scsi_remove_device+0x168/0x1a0 scsi_forget_host+0xa8/0xb0 scsi_remove_host+0x9b/0x150 sdebug_driver_remove+0x3d/0x140 [scsi_debug] device_remove+0x6f/0xb0 device_release_driver_internal+0x2c5/0x360 device_release_driver+0x12/0x20 bus_remove_device+0x1aa/0x270 device_del+0x2d4/0x640 device_unregister+0x18/0x70 sdebug_do_remove_host+0x138/0x180 [scsi_debug] scsi_debug_exit+0x45/0xd5 [scsi_debug] __do_sys_delete_module.constprop.0+0x210/0x320 __x64_sys_delete_module+0x1f/0x30 do_syscall_64+0x35/0x80 entry_SYSCALL_64_after_hwframe+0x44/0xaeOne of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?
Quoted 4 lines
Thanks,
Bart.
I reported/bisected for this issue last week, not sure if it helped.
https://lore.kernel.org/linux-block/CAHj4cs9OTm9sb_5fmzgz+W9OSLeVPKix3Yri856kqQVccwd_Mw@mail.gmail.com/T/#t
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 2 lines · Yi Zhang
I reported/bisected for this issue last week, not sure if it helped.
https://lore.kernel.org/linux-block/CAHj4cs9OTm9sb_5fmzgz+W9OSLeVPKix3Yri856kqQVccwd_Mw@mail.gmail.com/T/#t
That definitely helps :-) Since reverting the scsi_debug patch that has been identified as the root cause unbreaks blktests, I will post a revert of that patch.
Thanks,
Bart.
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?
That SYNCHRONIZE CACHE is a passthrough command sent on the request_queue and should not be affected by stopping all file system I/O.
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.
When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?
Quoted 8 lines
Quoted 2 lines
That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
Hi Bob,
After having taken a closer look at del_gendisk(), I agree with what Christoph wrote above. Please revert patch "scsi: scsi_debug: Address races following module load" locally when running blktests. See also https://lore.kernel.org/linux-scsi/[email protected]/T/#m47a23ffd5ce68b8183100444d6e711b6b4aba393.
Thanks,
Bart.
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?
Quoted 8 lines
Quoted 2 lines
That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
Quoted 20 lines · Yanjun Zhu
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?Quoted 8 lines
Quoted 2 lines
That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I know. I am trying to find out why. For performance reasons I very much want to make the xarray + rcu_locking patches work correctly. All the spinlock issues are now fixed in my tree. What is left is a race in the RDMA read retry code somewhere. I'll find it. In the process of chasing this I have found several real bugs and I suspect a few more are out there.
Bob
Quoted 20 lines · Yanjun Zhu
Quoted 10 lines · Bob Pearson
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?Quoted 8 lines
Quoted 2 lines
That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??
Quoted 6 lines
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.
I am not sure about this because wr NULL problem exists with xarray patches.
Please let us find the root cause of wr NULL.
This can make RXE more stable.
Zhu Yanjun
Quoted 27 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??Quoted 6 lines
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.
Quoted 8 lines
I am not sure about this because wr NULL problem exists with xarray patches.
Please let us find the root cause of wr NULL.
This can make RXE more stable.
Zhu Yanjun
You mean mr = NULL. And it is not happening in my tree. I have WARN_ONs looking for it and it isn't happening.
Quoted 30 lines · Bob Pearson
Quoted 29 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??Quoted 6 lines
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.
I am not sure about this because wr NULL problem exists with xarray patches.
Quoted 8 lines
Quoted 6 lines
Please let us find the root cause of wr NULL.
This can make RXE more stable.
Zhu Yanjun
You mean mr = NULL. And it is not happening in my tree. I have
WARN_ONslooking for it
Why you said that you can not reproduce this problem now?
Please check your mail. I remember that you can reproduce this wr NULL in your host.
Please focus on this problem. This can make RXE more stable. Let us find the root cause and fix this problem ASAP.
Thanks.
Zhu Yanjun
Quoted 1 line
and it isn't happening.
Quoted 35 lines · Yanjun Zhu
Quoted 34 lines · Bob Pearson
Quoted 33 lines · Yanjun Zhu
Quoted 18 lines · Bob Pearson
Quoted 17 lines · Yanjun Zhu
Quoted 16 lines · Bob Pearson
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Bart Van Assche
One of the functions in the above call stack is
sd_remove().sd_remove()callsdel_gendisk()just before callingsd_shutdown().sd_shutdown()submits the SYNCHRONIZE CACHE command. Indel_gendisk()I found the following comment: "Fail any new I/O". Do you agree that failing new I/O beforesd_shutdown()is called is wrong? Is there any other way to fix this than moving theblk_queue_start_drain()etc. calls out ofdel_gendisk()and into a new function?That SYNCHRONIZE CACHE is a passthrough command sent on the
request_queueand should not be affected by stopping all file system I/O.When I run check -q srp
all the test cases pass but each one stops for 3+ minutes at synchronize cache. The rxe device is still active until sync cache returns when the last QP and the PD are destroyed. It may be that the queues are blocked waiting for something else even though they have reported success??Quoted 6 lines
Quoted 4 lines
If you remove all the xarray patches and use the original source code. This will not occur.
Zhu Yanjun
I missed one other point. The 3 minute delay is actually not a rxe bug at all but was recently caused by a bad scsi patch which has since been reverted.
I am not sure about this because wr NULL problem exists with xarray patches.
Please let us find the root cause of wr NULL.
This can make RXE more stable.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Zhu Yanjun
You mean mr = NULL. And it is not happening in my tree. I have
WARN_ONslooking for itWhy you said that you can not reproduce this problem now?
Please check your mail. I remember that you can reproduce this wr NULL in your host.
Yes I did see it before but not with the new pool patches. The whole point of the patch series I have been working on for the past month was to clean up races in shutdown and cleanup code. These were demonstrated by colleagues here at hpe who are running lustre over soft roce. I looked at the shutdown code and saw that the whole design was flawed. We were using kref's to track reference counts on MRs and QPs etc. but always returning to rdma-core regardless of the reference count. rdma-core having no visibility to our references then deletes the object fairly soon after that while we are still trying to process late arriving packets. One of the features in my tree is to use wait_for_completion() in the return path to rdma-core and pause until all the references have been dropped.
I really don't see the point of debugging the old code because it is just wrong wrong wrong. It will never be stable under heavy load.
Bob
Quoted 9 lines
Please focus on this problem. This can make RXE more stable. Let us find the root cause and fix this problem ASAP.
Thanks.
Zhu YanjunQuoted 1 line
and it isn't happening.