handle_async_copy calling kzalloc under spinlock
Subsystems (2)
From bc0c9079b48d "NFS handle COPY reply CB_OFFLOAD call race":
+ spin_lock(&server->nfs_client->cl_lock);
+ list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids,
+ copies) {
+ if (memcmp(&res->write_res.stateid, ©->stateid,
+ NFS4_STATEID_SIZE))
+ continue;
+ found_pending = true; + list_del(©->copies); + break;
+ }
+ if (found_pending) {
+ spin_unlock(&server->nfs_client->cl_lock);
+ goto out;
+ }
copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);
At this point we're still holding cl_lock.
Best might be to allocate "copy" before taking the lock, then free it on any paths where we don't end up needing it.
--b.
A message that is not in the archive (sent off-list or not yet received): CAN-5tyGsDAd2hkaw5nONdS2TzRy1qK7xyFDpNeT7Jsd9ZrH7+g@mail.gmail.com
Quoted 22 lines · Olga Kornievskaia
Quoted 19 lines · J. Bruce Fields
From
bc0c9079b48d"NFS handle COPY replyCB_OFFLOADcall race":(quoted diff) +14
-0+ spin_lock(&server->nfs_client->cl_lock); + list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids, + copies) { + if (memcmp(&res->write_res.stateid, ©->stateid, + NFS4_STATEID_SIZE)) + continue; + found_pending = true; + list_del(©->copies); + break; + } + if (found_pending) { + spin_unlock(&server->nfs_client->cl_lock); + goto out; + } copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);
Quoted 8 lines
Quoted 7 lines
At this point we're still holding
cl_lock.Best might be to allocate "copy" before taking the lock, then free it on any
paths where we don't end up needing it.Thanks. I'll do that.
Thanks. And, I just noticed--nfs4_callback_offload has the same problem.
--b.
Quoted 30 lines · J. Bruce Fields
Quoted 29 lines · Olga Kornievskaia
Quoted 26 lines · J. Bruce Fields
From
bc0c9079b48d"NFS handle COPY replyCB_OFFLOADcall race":(quoted diff) +14
-0+ spin_lock(&server->nfs_client->cl_lock); + list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids, + copies) { + if (memcmp(&res->write_res.stateid, ©->stateid, + NFS4_STATEID_SIZE)) + continue; + found_pending = true; + list_del(©->copies); + break; + } + if (found_pending) { + spin_unlock(&server->nfs_client->cl_lock); + goto out; + } copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);At this point we're still holding
cl_lock.Best might be to allocate "copy" before taking the lock, then free it on any
paths where we don't end up needing it.
Quoted 4 lines
Quoted 1 line
Thanks. I'll do that.
Thanks. And, I just noticed--nfs4_callback_offload has the same problem.
By the way, I don't understand the create case in that code--if you get a CB_OFFLOAD without already having a matching copy stateid, shouldn't you just return an error and forget about it?
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
--b.
Quoted 32 lines · J. Bruce Fields
Quoted 30 lines · J. Bruce Fields
Quoted 29 lines · Olga Kornievskaia
Quoted 26 lines · J. Bruce Fields
From
bc0c9079b48d"NFS handle COPY replyCB_OFFLOADcall race":(quoted diff) +14
-0+ spin_lock(&server->nfs_client->cl_lock); + list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids, + copies) { + if (memcmp(&res->write_res.stateid, ©->stateid, + NFS4_STATEID_SIZE)) + continue; + found_pending = true; + list_del(©->copies); + break; + } + if (found_pending) { + spin_unlock(&server->nfs_client->cl_lock); + goto out; + } copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);At this point we're still holding
cl_lock.Best might be to allocate "copy" before taking the lock, then free it on any
paths where we don't end up needing it.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Thanks. I'll do that.
Thanks. And, I just noticed--nfs4_callback_offload has the same problem.
nfs4_callback_offload is where I changed it. I see now, handle_async_copy() in nfs42proc.c also has it.
Quoted 3 lines
By the way, I don't understand the create case in that code--if you get a
CB_OFFLOADwithout already having a matching copy stateid, shouldn't you just return an error and forget about it?
Then how does the copy knows not to go wait for the callback? Copy checks the pending_callback list to see if received a callback. If not, it puts itself on the copy list and goes to sleep. The callback, checks the copy list and if it finds a copy signals it, if not it puts itself on the pending_callback list. a lock is held over checking one list and putting yourself on the other.
Quoted 2 lines
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
I guess EIO or ENOMEM might be better. But I don't think this error gets returned anywhere to the main process.
Quoted 2 lines
--b.
A message that is not in the archive (sent off-list or not yet received): CAN-5tyHX3h6TXJhFeZPGZvAWXXxwaxpAkZtRNV9+L8m5xJ3fVw@mail.gmail.com
Quoted 7 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
Then how does the copy knows not to go wait for the callback? Copy checks the
pending_callbacklist to see if received a callback. If not, it puts itself on the copy list and goes to sleep. The callback, checks the copy list and if it finds a copy signals it, if not it puts itself on thepending_callbacklist. a lock is held over checking one list and putting yourself on the other.
OK, apologies, I don't really understand those data structures yet, but something seems wrong to me.
Under what circumstances could we recieve a CB_OFFLOAD without having started the corresponding copy already?
And shouldn't CB_OFFLOAD be returning bad_stateid in the case it doesn't recognize the given stateid? It looks like the allocation failure is the *only* way we'll return an error on CB_OFFLOAD, and that seems wrong.
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
I guess EIO or ENOMEM might be better. But I don't think this error gets returned anywhere to the main process.
Wait. It is returning SERVERFAULT because it's the callback server replying back to the server's
CB_RECALLcall and I believe SERVERFAULT is the appropriate error here. NFS doesn't have ENOMEM error.
We could return DELAY if we think it might be worth the server trying the CB_RECALL again. (That's what nfsd usually returns on allocation failures. I don't know if that's really ideal.)
--b.
Quoted 10 lines · J. Bruce Fields
Quoted 7 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
Then how does the copy knows not to go wait for the callback? Copy checks the
pending_callbacklist to see if received a callback. If not, it puts itself on the copy list and goes to sleep. The callback, checks the copy list and if it finds a copy signals it, if not it puts itself on thepending_callbacklist. a lock is held over checking one list and putting yourself on the other.
Quoted 5 lines
OK, apologies, I don't really understand those data structures yet, but something seems wrong to me.
Under what circumstances could we recieve a
CB_OFFLOADwithout having started the corresponding copy already?
It can receive a CB_OFFLOAD before it receives a reply to the COPY. It's possible and I can trigger it during testing when doing a really short copy. The copy is done and callback thread sends a reply. CB_OFFLOAD call and COPY reply can be switched on the server or on the processing on the client.
Quoted 3 lines
And shouldn't
CB_OFFLOADbe returningbad_stateidin the case it doesn't recognize the given stateid?
It could but what should the server do in this case. I would imagine it wouldn't do anything. There is nothing it can do. So now we have a copy that send the call and is going to wait on the reply which will never come as the 1st one came and we rejected it and now copy will wait forever.
Please describe what "is wrong" with the current implementation. I believe it provide a reasonable solution to the race condition.
Quoted 3 lines
It looks like the allocation failure is
the *only* way we'll return an error onCB_OFFLOAD, and that seems wrong.
Yes it is the only error we currently return. Do you see any other useful errors that a client should return (and would be useful to handle on the server). I don't see any need for any more complications.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
I guess EIO or ENOMEM might be better. But I don't think this error gets returned anywhere to the main process.
Quoted 7 lines
Quoted 3 lines
Wait. It is returning SERVERFAULT because it's the callback server replying back to the server's
CB_RECALLcall and I believe SERVERFAULT is the appropriate error here. NFS doesn't have ENOMEM error.We could return DELAY if we think it might be worth the server trying the
CB_RECALLagain. (That's what nfsd usually returns on allocation failures. I don't know if that's really ideal.)
If the client had any smarts to say correct this error that would be useful to return but this is not the case. I don't believe there is a
Quoted 14 lines · Olga Kornievskaia
Quoted 13 lines · J. Bruce Fields
Quoted 7 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
Then how does the copy knows not to go wait for the callback? Copy checks the
pending_callbacklist to see if received a callback. If not, it puts itself on the copy list and goes to sleep. The callback, checks the copy list and if it finds a copy signals it, if not it puts itself on thepending_callbacklist. a lock is held over checking one list and putting yourself on the other.OK, apologies, I don't really understand those data structures yet, but something seems wrong to me.
Quoted 8 lines
Quoted 2 lines
Under what circumstances could we recieve a
CB_OFFLOADwithout having started the corresponding copy already?It can receive a
CB_OFFLOADbefore it receives a reply to the COPY. It's possible and I can trigger it during testing when doing a really short copy. The copy is done and callback thread sends a reply.CB_OFFLOADcall and COPY reply can be switched on the server or on the processing on the client.
That race is discussed in
https://tools.ietf.org/html/rfc5661#section-2.10.6.3 and is supposed to be dealt with by using referring triples and/or returning DELAY.
Quoted 11 lines
Quoted 2 lines
And shouldn't
CB_OFFLOADbe returningbad_stateidin the case it doesn't recognize the given stateid?It could but what should the server do in this case. I would imagine it wouldn't do anything. There is nothing it can do. So now we have a copy that send the call and is going to wait on the reply which will never come as the 1st one came and we rejected it and now copy will wait forever.
Please describe what "is wrong" with the current implementation. I believe it provide a reasonable solution to the race condition.
Looks like a server that sends bad stateids in callbacks could cause you to allocate something that will never get freed.
--b.
Quoted 26 lines
Quoted 3 lines
It looks like the allocation failure is
the *only* way we'll return an error onCB_OFFLOAD, and that seems wrong.Yes it is the only error we currently return. Do you see any other useful errors that a client should return (and would be useful to handle on the server). I don't see any need for any more complications.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
I guess EIO or ENOMEM might be better. But I don't think this error gets returned anywhere to the main process.
Quoted 7 lines
Quoted 3 lines
Wait. It is returning SERVERFAULT because it's the callback server replying back to the server's
CB_RECALLcall and I believe SERVERFAULT is the appropriate error here. NFS doesn't have ENOMEM error.We could return DELAY if we think it might be worth the server trying the
CB_RECALLagain. (That's what nfsd usually returns on allocation failures. I don't know if that's really ideal.)If the client had any smarts to say correct this error that would be useful to return but this is not the case. I don't believe there is a
Quoted 25 lines · J. Bruce Fields
Quoted 14 lines · Olga Kornievskaia
Quoted 13 lines · J. Bruce Fields
Quoted 7 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
Then how does the copy knows not to go wait for the callback? Copy checks the
pending_callbacklist to see if received a callback. If not, it puts itself on the copy list and goes to sleep. The callback, checks the copy list and if it finds a copy signals it, if not it puts itself on thepending_callbacklist. a lock is held over checking one list and putting yourself on the other.OK, apologies, I don't really understand those data structures yet, but something seems wrong to me.
Quoted 8 lines
Quoted 2 lines
Under what circumstances could we recieve a
CB_OFFLOADwithout having started the corresponding copy already?It can receive a
CB_OFFLOADbefore it receives a reply to the COPY. It's possible and I can trigger it during testing when doing a really short copy. The copy is done and callback thread sends a reply.CB_OFFLOADcall and COPY reply can be switched on the server or on the processing on the client.
Quoted 3 lines
That race is discussed in
https://tools.ietf.org/html/rfc5661#section-2.10.6.3 and is supposed to be dealt with by using referring triples and/or returning DELAY.
I believe those are suggestions and not mandates? A client can't rely that the server will implement referring sequence information. Sending "delay" to the server might be an option but it's an option that most like will interfere with performance as well?
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
And shouldn't
CB_OFFLOADbe returningbad_stateidin the case it doesn't recognize the given stateid?It could but what should the server do in this case. I would imagine it wouldn't do anything. There is nothing it can do. So now we have a copy that send the call and is going to wait on the reply which will never come as the 1st one came and we rejected it and now copy will wait forever.
Quoted 5 lines
Quoted 2 lines
Please describe what "is wrong" with the current implementation. I believe it provide a reasonable solution to the race condition.
Looks like a server that sends bad stateids in callbacks could cause you to allocate something that will never get freed.
I thought the philosophy was that client shouldn't be coded to a broken server. If needed, we can later on add a cleanup thread that goes thru the list and removes really old entries.
Quoted 29 lines
--b.
Quoted 26 lines
Quoted 3 lines
It looks like the allocation failure is
the *only* way we'll return an error onCB_OFFLOAD, and that seems wrong.Yes it is the only error we currently return. Do you see any other useful errors that a client should return (and would be useful to handle on the server). I don't see any need for any more complications.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
I also wonder if SERVERFAULT is really the best error for a memory allocation failure there.
I guess EIO or ENOMEM might be better. But I don't think this error gets returned anywhere to the main process.
Quoted 7 lines
Quoted 3 lines
Wait. It is returning SERVERFAULT because it's the callback server replying back to the server's
CB_RECALLcall and I believe SERVERFAULT is the appropriate error here. NFS doesn't have ENOMEM error.We could return DELAY if we think it might be worth the server trying the
CB_RECALLagain. (That's what nfsd usually returns on allocation failures. I don't know if that's really ideal.)If the client had any smarts to say correct this error that would be useful to return but this is not the case. I don't believe there is a
Quoted 9 lines · Olga Kornievskaia
Quoted 3 lines · J. Bruce Fields
That race is discussed in
https://tools.ietf.org/html/rfc5661#section-2.10.6.3 and is supposed to be dealt with by using referring triples and/or returning DELAY.I believe those are suggestions and not mandates? A client can't rely that the server will implement referring sequence information. Sending "delay" to the server might be an option but it's an option that most like will interfere with performance as well?
Yes, I suppose the server either needs to implement referring triples or retry pretty aggressively.
(By the way, I wonder if the server should always do synchronous copies for copies smaller than a certain threshhold. Might be hard to choose the threshhold, though.)
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 2 lines
And shouldn't
CB_OFFLOADbe returningbad_stateidin the case it doesn't recognize the given stateid?It could but what should the server do in this case. I would imagine it wouldn't do anything. There is nothing it can do. So now we have a copy that send the call and is going to wait on the reply which will never come as the 1st one came and we rejected it and now copy will wait forever.
Please describe what "is wrong" with the current implementation. I believe it provide a reasonable solution to the race condition.
Quoted 6 lines
Quoted 2 lines
Looks like a server that sends bad stateids in callbacks could cause you to allocate something that will never get freed.
I thought the philosophy was that client shouldn't be coded to a broken server. If needed, we can later on add a cleanup thread that goes thru the list and removes really old entries.
I suppose so.
I don't know, this design still makes me pretty uncomfortable, but I guess I haven't come up with a strong reason it couldn't work.
--b.