BDEV-IO Lifecycle - Need your input.

Hi Ben,

Your proposal<https://review.gerrithub.io/#/c/spdk/spdk/+/386166/> to interface with the backend to acquire and release buffers is good. You have accurately stated that the challenge is in developing intuitive semantics. And that has been my struggle. To me, there are two problem statements;

1. It is expected that bdev_io pool is sized correctly so the call to get bdev_io succeeds. Failure to acquire bdev_io will result in DEVICE-ERROR. The transport is already capable of handling temporary memory failures by moving the request to PENDING-Queue. Hence the proposal to change the bdev_io lifecycle and perhaps connect it with the spdk_nvmf_request object. Thus all buffer needs are addressed at the beginning of I/O request.
2. I/O Data buffers are sourced and managed by the backend. One of the challenges I see with your proposed interface is the lack of details like if resource is being acquired for a READ operation or WRITE operation. The handling is quite different in each case. Since bdev_io->type is overloaded the type of I/O operation is lost. I suppose one can cast the cb_arg (nvmf_request) and then proceed. Also, the bdev_io should be present to RELEASE the buffer. Zero copy semantics warrants that data buffer stays until controller-to-host has occurred. In other words, bdev_io lives till REQUEST come to COMPLETE state.

What are your thoughts in introducing spdk_bdev_init() and spdk_bdev_fini() as an alternative approach to extend the lifecyle of bdev_io and allow data buffer management via bdev fn_table ?

I hope I’m making sense…

Thanks,
Srikanth

From: Walker, Benjamin <benjamin.walker(a)intel.com>
Sent: Friday, May 11, 2018 12:28 PM
To: Harris, James R <james.r.harris(a)intel.com>; Kaligotla, Srikanth <Srikanth.Kaligotla(a)netapp.com>
Cc: raju.gottumukkala(a)broadcom.com; Meneghini, John <John.Meneghini(a)netapp.com>; Rodriguez, Edwin <Ed.Rodriguez(a)netapp.com>; Pai, Madhu <Madhusudan.Pai(a)netapp.com>; NGC-john.barnard-broadcom.com <john.barnard(a)broadcom.com>
Subject: Re: BDEV-IO Lifecycle - Need your input.

Hi Srikanth,

Yes - we'll need to introduce some way to acquire and release buffers from the bdev layer earlier on in the state machine that processes an NVMe-oF request. I've had this patch out for review for several months as a proposal for this scenario:

https://review.gerrithub.io/#/c/spdk/spdk/+/386166/

It doesn't pass the tests - it's just a proposal for the interface. 90% of the challenge here is in developing intuitive semantics.

Thanks,
Ben

P.S. This is the kind of discussion that would fit perfectly on the mailing list.

On Wed, 2018-05-09 at 20:35 +0000, Kaligotla, Srikanth wrote:
Hi Ben, Hi James,

I would like to solicit opinions on the lifecycle for bdev-io resource object. Attached is the image of RDMA state machine in its current implementation. When the REQUEST enters the NEED-BUFFER state, the buffers necessary for carrying out I/O operation are allocated/acquired from the memory pool. An instance of BDEV-IO comes into existence after REQUEST reaches READY-TO-EXECUTE state. The BDEV-IO is teared down as soon the backend returns. From a BDEV perspective, BDEV-IO is simply a translation unit that facilitates I/O buffers from the backend. The driver context embedded within bdev_io holds great deal of information pertaining to the I/O under execution. It assists in error handling, dereferencing the buffers upon I/O completion and in abort handling. In summary, the bdev_io stays alive until request has come to COMPLETE state. I’d like to hear peoples thoughts in introducing the plumbing to acquire BDEV-IO resource in REQUEST-NEED-BUFFER state and release it in REQUEST-COMPLETE state. I will shortly have patch available for review that introduces spdk_bdev_init and spdk_bdev_fini which in turn invokes the corresponding bdev fn_table to initialize/cleanup.

I wanted to use this email to communicate our intent and solicit your feedback. We have a working implementation of the above proposal and prior to pushing it upstream for review would like to hear your thoughts. These proposed changes to upstream are a result of FC transport work in collaboration with the Broadcom team who are also copied to this mail. Myself and John will be at SPDK Dev conference and if required we can elaborate further on this proposal.

Thanks,
Srikanth