SRP: ProfitBricks publishes its SRP Initiator patches
Hello everyone,
I'm very proud to announce that we finally publish our SRP initiator patches we've been working on for quite some time now.
In the first step we publish our way of failing IO fast as we've noticed that the way Bart Van Assche does that in his GitHub repository doesn't match our requirements completely.
His repo: https://github.com/bvanassche/ib_srp-backport
Our repo: https://github.com/sriemer/ib_srp
We want to fail IO fast in exactly the time we configure. With our patches this works (or please tell us why not). We provide you with full test descriptions and related shell scripts. Everything is done with as little dependencies as possible.
The shell scripts can also be very useful to show how to configure and use SRP with sysfs only. This is why I've added the scst-devel mailing list here. We want to be as close as possible to the kernel.
We want to combine efforts here and to get valuable feedback from you all. Evaluation, testing, criticism, comments, etc more than welcome! Hopefully, we can get a really cool solution into the mainline together! This would make my job as the maintainer for the ProfitBricks host kernels a lot easier! ;-)
You'll notice: We've already adapted patches from Bart and parts of his patches. So it is only fair to publish our patches as well. :-)
It is the same as with Bart's patches: This can't be used for production, yet. Our patches don't have the reconnect for now. Ideas how to implement that on top are welcome. Just send your patches directly to me! :-)
Please further notice: There is a major bug in the upstream multipath-tools. These read sysfs files cached which leads to IO on offline devices. We've fixed it for us and publish the fix for you as well. :-)
Git repo:
https://github.com/sriemer/multipath-tools
Thank you so much for your help in the past and in the future as well! Thanks for the patience and reading this!
We'll continue publishing our SRP patches relevant for the mainline. If you want to meet me or ProfitBricks in person, we'll have a booth on LinuxTag in Berlin/Germany. I'll have a technical talk there about SRP:
http://www.linuxtag.org/2013/en/program/thursday-may-23-2013.html?eventid=208
Cheers,
Sebastian
Signature · 13 lines
Sebastian Riemer Linux Kernel Developer - Storage ProfitBricks GmbH • Greifswalder Str. 207 • 10405 Berlin, Germany www.profitbricks.com • [email protected] Sitz der Gesellschaft: Berlin Registergericht: Amtsgericht Charlottenburg, HRB 125506 B Geschäftsführer: Andreas Gauger, Achim Weiss -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello!
What is main difference between bvanassche repo and sriemer ?
2013/4/12 Sebastian Riemer <[email protected]>:
Quoted 69 lines
Hello everyone,
I'm very proud to announce that we finally publish our SRP initiator patches we've been working on for quite some time now.
In the first step we publish our way of failing IO fast as we've noticed that the way Bart Van Assche does that in his GitHub repository doesn't match our requirements completely.
His repo: https://github.com/bvanassche/ib_srp-backport
Our repo: https://github.com/sriemer/ib_srpWe want to fail IO fast in exactly the time we configure. With our patches this works (or please tell us why not). We provide you with full test descriptions and related shell scripts. Everything is done with as little dependencies as possible.
The shell scripts can also be very useful to show how to configure and use SRP with sysfs only. This is why I've added the scst-devel mailing list here. We want to be as close as possible to the kernel.
We want to combine efforts here and to get valuable feedback from you all. Evaluation, testing, criticism, comments, etc more than welcome! Hopefully, we can get a really cool solution into the mainline together! This would make my job as the maintainer for the ProfitBricks host kernels a lot easier! ;-)
You'll notice: We've already adapted patches from Bart and parts of his patches. So it is only fair to publish our patches as well. :-)
It is the same as with Bart's patches: This can't be used for production, yet. Our patches don't have the reconnect for now. Ideas how to implement that on top are welcome. Just send your patches directly to me! :-)
Please further notice: There is a major bug in the upstream multipath-tools. These read sysfs files cached which leads to IO on offline devices. We've fixed it for us and publish the fix for you as well. :-)
Git repo:
https://github.com/sriemer/multipath-toolsThank you so much for your help in the past and in the future as well! Thanks for the patience and reading this!
We'll continue publishing our SRP patches relevant for the mainline. If you want to meet me or ProfitBricks in person, we'll have a booth on LinuxTag in Berlin/Germany. I'll have a technical talk there about SRP:
http://www.linuxtag.org/2013/en/program/thursday-may-23-2013.html?eventid=208
Cheers,
Sebastian--
Sebastian Riemer
Linux Kernel Developer - StorageProfitBricks GmbH • Greifswalder Str. 207 • 10405 Berlin, Germany www.profitbricks.com • [email protected]
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B Geschäftsführer: Andreas Gauger, Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 7 lines
Vasiliy Tolstov, e-mail: [email protected] jabber: [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CACaajQs0PswMb5KhpSkE1=_Sd1SwX2BznZP9x1FCkEk6ukJYrA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 1 line · Vasiliy Tolstov
What is main difference between bvanassche repo and sriemer ?
Good question. As soon as I have the time I will try to find a single approach that works for everyone and post a new patch series for review on the linux-rdma mailing list such that these can be included in the mainline kernel.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CACaajQtQk0YJ3g0G5PyOue0qG3SzhaW3qmXj_TLqYDVqzKwJXQ@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CACaajQtQk0YJ3g0G5PyOue0qG3SzhaW3qmXj_TLqYDVqzKwJXQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Hi Vasiliy,
sorry for the late reply! I was ill last week.
The main difference so far is that my patches are much easier to understand as I don't provide back-porting and also don't put performance improvements in between the stability fixes. I provide full test cases + scripts which makes it simple to check if you've got the expected/same behavior. Everything is straight forward and completely described in the commit messages.
My work focuses on using ib_srp without the srp_tools. Just fast IO failing + reconnect. No performance improvements. We gradually find compromises to get a cool solution. We've e.g. agreed on detecting connection loss upon the QP timeout only.
Today I'll release our own automatic reconnect.
Sorry Bart, but a reconnect with just the commit message "IB/srp: Add kernel-level transport layer recovery" and no further description isn't very trustworthy for me. I also wonder why you need so much locking.
Cheers,
Sebastian
Quoted 10 lines · Vasiliy Tolstov
03.05.2013 15:45 пользователь "Bart Van Assche" <[email protected] <mailto:[email protected]>> написал:
Quoted 7 lines · Vasiliy Tolstov
Quoted 2 lines
What is main difference between bvanassche repo and sriemer ?
Good question. As soon as I have the time I will try to find a single
Quoted 9 lines
approach that works for everyone and post a new patch series for review on the linux-rdma mailing list such that these can be included in the mainline kernel.
Quoted 2 lines
Bart.
Thats very good! Thans! I'm already using 3.8 with you patches and all works very stable.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Sebastian Riemer
Sorry Bart, but a reconnect with just the commit message "IB/srp: Add kernel-level transport layer recovery" and no further description isn't very trustworthy for me. I also wonder why you need so much locking.
Hello Sebastian,
There is a very good reason for all that locking: during device removal the rport structure is freed before SCSI host removal finished. So when converting a SCSI host pointer into an rport pointer protection is needed against device removal. The current approach in the ib_srp-backport repository is to realize that protection via locking. I'm currently reworking that approach into protection via reference counting such that it is guaranteed that the rport remains to exist until scsi_remove_device() has finished. That will allow to eliminate several locking statements.
Sorry for the terse patch descriptions - I had not expected that anyone would try to understand the patches ib_srp-backport repository and use these as a basis for further work.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
FYI: I've released version 0.6 of my SRP patches today.
The automatic reconnect is included now. The tests for that will follow in the next version. But we already did quite intensive testing for that.
Hard reboot and also soft reboot of the target are possible with that reconnect. It just reconnects and everything is fine again.
With soft reboot I mean: disabling the target, removing the exports, rebooting, exporting the same LUNs, re-enabling the target.
It also has an automatic mechanism to reduce the possibility of a DDoS attack reconnect. It automatically reconnects at different intervals.
Check it out:
https://github.com/sriemer/ib_srp
Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
if i need faster reconnects and ability to close session from initiator side under qlogic hardware, does it possible? Or this patches only covers mallanox cards?
2013/5/8 Sebastian Riemer <[email protected]>:
Quoted 19 lines
FYI: I've released version 0.6 of my SRP patches today.
The automatic reconnect is included now. The tests for that will follow in the next version. But we already did quite intensive testing for that.
Hard reboot and also soft reboot of the target are possible with that reconnect. It just reconnects and everything is fine again.
With soft reboot I mean: disabling the target, removing the exports, rebooting, exporting the same LUNs, re-enabling the target.
It also has an automatic mechanism to reduce the possibility of a DDoS attack reconnect. It automatically reconnects at different intervals.
Check it out:
https://github.com/sriemer/ib_srpCheers,
Sebastian
Signature · 7 lines
Vasiliy Tolstov, e-mail: [email protected] jabber: [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CACaajQun61QAM0qqwadWECSFO7UxUaOrdO1-0vGLPQBjdwdMyw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Of cause, also Qlogic HCAs can be used.
But please note that there is no back-port in my patches to make them better readable. If you like my patches, then we can talk about how to back-port them to a specific kernel version.
Quoted 4 lines · Vasiliy Tolstov
if i need faster reconnects and ability to close session from initiator side under qlogic hardware, does it possible? Or this patches only covers mallanox cards?
Quoted 23 lines
2013/5/8 Sebastian Riemer <[email protected]>:
Quoted 19 lines
FYI: I've released version 0.6 of my SRP patches today.
The automatic reconnect is included now. The tests for that will follow in the next version. But we already did quite intensive testing for that.
Hard reboot and also soft reboot of the target are possible with that reconnect. It just reconnects and everything is fine again.
With soft reboot I mean: disabling the target, removing the exports, rebooting, exporting the same LUNs, re-enabling the target.
It also has an automatic mechanism to reduce the possibility of a DDoS attack reconnect. It automatically reconnects at different intervals.
Check it out:
https://github.com/sriemer/ib_srpCheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Vasiliy Tolstov
if i need faster reconnects and ability to close session from initiator side under qlogic hardware, does it possible? Or this patches only covers mallanox cards?
The ability to close a session from the initiator side went upstream in kernel 3.8 (/sys/class/srp_remote_ports/port-<h>:<n>/delete). Regarding faster reconnects: please keep in mind that after a cable pull it can easily takes 20 seconds before link training and initialization by the subnet manager have finished. It's not possible to make an initiator reconnect in less time than what the hardware and subnet manager need to bring the link back.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
2013/5/14 Bart Van Assche <[email protected]>:
Quoted 7 lines
The ability to close a session from the initiator side went upstream in kernel 3.8 (/sys/class/srp_remote_ports/port-<h>:<n>/delete). Regarding faster reconnects: please keep in mind that after a cable pull it can easily takes 20 seconds before link training and initialization by the subnet manager have finished. It's not possible to make an initiator reconnect in less time than what the hardware and subnet manager need to bring the link back.
Thanks. What about close session from target side? For example i need to close the srp session and block all access from specific initiator?
--
Vasiliy Tolstov,
e-mail: [email protected]
jabber: [email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CACaajQt2V1Y-fQFmXyyAJk57KahbMwBqvEGoF7sug5F1hYyWGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 10 lines · Vasiliy Tolstov
2013/5/14 Bart Van Assche <[email protected]>:
Quoted 7 lines
The ability to close a session from the initiator side went upstream in kernel 3.8 (/sys/class/srp_remote_ports/port-<h>:<n>/delete). Regarding faster reconnects: please keep in mind that after a cable pull it can easily takes 20 seconds before link training and initialization by the subnet manager have finished. It's not possible to make an initiator reconnect in less time than what the hardware and subnet manager need to bring the link back.
Quoted 2 lines
Thanks. What about close session from target side? For example i need to close the srp session and block all access from specific initiator?
AFAIK the session is blocked as long as an initiator is connected. The only possibility besides disconnecting the initiators is to disable the target completely. Then, it sends a DREQ (disconnection request) to the initiators. These know then that the target is disconnected and send a DREP (disconnection reply). In our patches we also activate the reconnect in this situation as we do that to orderly reboot a storage system (e.g. due to an issue). The storage system comes up again, exports the same volumes and the initiators can reconnect.
Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Vasiliy Tolstov
Thanks. What about close session from target side? For example i need to close the srp session and block all access from specific initiator?
The traditional approach to block access from a specific initiator is to modify the LUN masking configuration at the target side dynamically. More information about LUN masking can be found in the scst.conf man page and in the srpt/README file. I will make a patch available for ib_srpt that allows to close a single session at a time.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
2013/5/15 Bart Van Assche <[email protected]>:
Quoted 5 lines
The traditional approach to block access from a specific initiator is to modify the LUN masking configuration at the target side dynamically. More information about LUN masking can be found in the scst.conf man page and in the srpt/README file. I will make a patch available for
ib_srptthat allows to close a single session at a time.
Thanks for answers and Thanks for patch, i'm try it.
--
Vasiliy Tolstov,
e-mail: [email protected]
jabber: [email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html