Sockets as proper objects and buffers with owners
Subsystems (3)
This is an update of my previous set to make sockets proper objhash citizens and use that functionality to attach owners to socket buffers for proper recording of the sender post-restart.
I still haven't done the collect() operation, but am sending this out to get feedback on the updated last patch which changes the sequence of events required to restore a socket.
A message that is not in the archive (sent off-list or not yet received): 1251915760-20118-1-git-send-email-danms-r/[email protected]
PATCH 1/3 Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
This changes the checkpoint/restart procedure for sockets a bit. The socket file header is now checkpointed separately from the socket itself, which allows us to checkpoint a socket without arriving at it from a file descriptor. Thus, most sockets will be checkpointed as a result of processing the file table, calling sock_file_checkpoint(fd), which in turn calls checkpoint_obj(socket).
However, we may arrive at some sockets while checkpointing other objects, such as the other end of an AF_UNIX socket with buffers in flight. This patch just opens that door, which is utilized by the next patch.
Changes in v2:
- If we attempt to checkpoint an orphan socket, create a struct socket to adopt it for the purposes of the checkpoint
--- checkpoint/objhash.c | 2 + include/linux/checkpoint_hdr.h | 6 +- include/net/sock.h | 2 + net/checkpoint.c | 140 +++++++++++++++++++++++++++++++-------- net/unix/checkpoint.c | 3 +- 5 files changed, 120 insertions(+), 33 deletions(-)
checkpoint/objhash.c +2 -0
@@ -381,6 +381,8 @@ static struct ckpt_obj_ops ckpt_obj_ops[] = { .obj_type = CKPT_OBJ_SOCK, .ref_drop = obj_sock_drop, .ref_grab = obj_sock_grab, + .checkpoint = checkpoint_sock, + .restore = restore_sock, }, };
include/linux/checkpoint_hdr.h +4 -2
@@ -70,6 +70,7 @@ enum {CKPT_HDR_USER ,CKPT_HDR_GROUPINFO ,CKPT_HDR_TASK_CREDS , + CKPT_HDR_SOCKET,/* 201-299: reserved for arch-dependent */
@@ -368,7 +369,8 @@ struct ckpt_hdr_file_pipe { } __attribute__((aligned(8)));/* socket */ -struct ckpt_socket { +struct ckpt_hdr_socket { +struct ckpt_hdr h;struct {/* struct socket */ __u64 flags;__u8 state;
@@ -428,7 +430,7 @@ struct ckpt_hdr_socket_unix {struct ckpt_hdr_file_socket {struct ckpt_hdr_file common; -struct ckpt_socket socket; +__s32 sock_objref; } __attribute__((aligned (8 )));struct ckpt_hdr_utsns {
include/net/sock.h +2 -0
@@ -1644,6 +1644,8 @@ extern __u32 sysctl_rmem_default;/* Checkpoint/Restart Functions */ struct ckpt_ctx ;struct ckpt_hdr_file ; +extern int checkpoint_sock (struct ckpt_ctx *ctx,void *ptr); +extern void *restore_sock (struct ckpt_ctx *ctx);extern int sock_file_checkpoint (struct ckpt_ctx *ctx,struct file *file);extern struct file *sock_file_restore (struct ckpt_ctx *ctx,struct ckpt_hdr_file *h);
net/checkpoint.c +111 -29
202 lines not on this page · raw message
net/unix/checkpoint.c +1 -2
@@ -57,7 +57,6 @@ static int unix_write_cwd(struct ckpt_ctx *ctx,int unix_checkpoint (struct ckpt_ctx *ctx,struct socket *sock) {struct unix_sock *sk =unix_sk (sock->sk); -struct unix_sock *pr =unix_sk (sk->peer);struct ckpt_hdr_socket_unix *un;int new;int ret = -ENOMEM ;
@@ -86,7 +85,7 @@ int unix_checkpoint(struct ckpt_ctx *ctx, struct socket *sock) goto out;if (sk->peer) - un->peer =ckpt_obj_lookup_add (ctx, pr,CKPT_OBJ_SOCK , &new); + un->peer =checkpoint_obj (ctx, sk->peer,CKPT_OBJ_SOCK );else un->peer =0 ;
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1251915760-20118-2-git-send-email-danms-r/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
Quoted 11 lines · Dan Smith
This changes the checkpoint/restart procedure for sockets a bit. The socket file header is now checkpointed separately from the socket itself, which allows us to checkpoint a socket without arriving at it from a file descriptor. Thus, most sockets will be checkpointed as a result of processing the file table, calling
sock_file_checkpoint(fd), which in turn callscheckpoint_obj(socket).However, we may arrive at some sockets while checkpointing other objects, such as the other end of an
AF_UNIXsocket with buffers in flight. This patch just opens that door, which is utilized by the next patch.
Quoted 5 lines
Changes in v2:
- If we attempt to checkpoint an orphan socket, create a struct socket to adopt it for the purposes of the checkpointSigned-off-by Dan Smith <danms-r/[email protected]>
Nice. See comments inline.
Quoted 8 lines
--- checkpoint/objhash.c | 2 + include/linux/checkpoint_hdr.h | 6 +- include/net/sock.h | 2 + net/checkpoint.c | 140 +++++++++++++++++++++++++++++++-------- net/unix/checkpoint.c | 3 +- 5 files changed, 120 insertions(+), 33 deletions(-)
[...]
net/checkpoint.c int sock_file_checkpoint(struct ckpt_ctx *ctx, struct file *file) · line 481
Quoted 12 lines
net/checkpoint.c +9
-1-int sock_file_checkpoint (struct ckpt_ctx *ctx,struct file *file) +static int __do_sock_checkpoint (struct ckpt_ctx *ctx,struct sock *sk) { +static int do_sock_checkpoint (struct ckpt_ctx *ctx,struct sock *sk) +{ +struct socket *sock; +int ret; + +if (sk->sk_socket) +return __do_sock_checkpoint (ctx, sk); +
I wonder if temporarily grafting the @sk is better than explicitly testing for sk->sk_socket everywhere else ? (in some places, the test already exists, so if we keep this approach, it should be removed).
As it is, it simplifies the code. However, how can we be certain that grafting a dead socket doesn't break any assumptions elsewhere ? For instance, unix_sock_destructor() will spit a warning if it finds sk->sk_socket != NULL. There may be other places (per protocol ?).
The alternative is to always pass around an 'sk' instead of sometimes a 'sock', and to skip parts of the checkpoint (and restart) entirely for dead sockets.
[...]
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
OL> As it is, it simplifies the code. However, how can we be certain OL> that grafting a dead socket doesn't break any assumptions OL> elsewhere ? For instance, unix_sock_destructor() will spit a OL> warning if it finds
sk-> sk_socket != NULL. There may be other places (per protocol ?).
I think that in most cases the grafted struct socket is short-lived and therefore shouldn't be a problem with something like the destructor.
I definitely prefer assuming we have an sk->sk_socket elsewhere and will be happy to remove the other checks, unless there's a compelling reason not to.
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
In reply to a message not in the archiveQuoted 5 lines · Dan Smith
net/checkpoint.c +5
-0+static int do_sock_checkpoint (struct ckpt_ctx *ctx,struct sock *sk) +{ +struct socket *sock; +int ret; +
net/checkpoint.c int sock_file_checkpoint(struct ckpt_ctx *ctx, struct file *file) · line 488
Quoted 10 lines
net/checkpoint.c +10
-0+if (sk->sk_socket) +return __do_sock_checkpoint (ctx, sk); + +/* Temporarily adopt this orphan socket */ + ret =sock_create (sk->sk_family, sk->sk_type,0 , &sock); +if (ret <0 ) +return ret; +sock_graft (sk, sock); + + ret =__do_sock_checkpoint (ctx, sk);
I'm sure I sound like an idiot, but... at restore, a socket will be created for sk now. Is that a problem? I don't see where sk_free() will cause that sock to be freed, and you are not attaching it do a file whose close would cause it to be released...
-serge
A message that is not in the archive (sent off-list or not yet received): 20090903054058.GA7189-r/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
SH> I'm sure I sound like an idiot, but... at restore, a socket will SH> be created for sk now. Is that a problem? I don't see where SH> sk_free() will cause that sock to be freed, and you are not SH> attaching it do a file whose close would cause it to be SH> released...
In the regular socket code, the reference that is taken during allocation is assumed to be for the owning object (the file). So, they don't take another ref when they actually attach it to a file.
The objhash assumes that for all objects on restore, the allocation routine for the object has already incremented the reference count for it as the owner. If you look at restore_obj() it drops the ref count right after stuffing the object in the hash because the act of insertion grabs a reference. On checkpoint, this is the reference for the hash. On restore, it would be redundant because it is the first owner (not a file, etc).
My code takes *another* reference when it attaches to a file, if appropriate. In the case of the adopted socket, only the objhash holds a reference to it, so unless it was joined to a peer, it will be freed when the objhash does its final obj_ref_drop() at tear-down time.
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 87eiqoyvkz.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
Quoted 18 lines · Dan Smith
SH> I'm sure I sound like an idiot, but... at restore, a socket will SH> be created for sk now. Is that a problem? I don't see where SH>
sk_free()will cause that sock to be freed, and you are not SH> attaching it do a file whose close would cause it to be SH> released...In the regular socket code, the reference that is taken during allocation is assumed to be for the owning object (the file). So, they don't take another ref when they actually attach it to a file.
The objhash assumes that for all objects on restore, the allocation routine for the object has already incremented the reference count for it as the owner. If you look at
restore_obj()it drops the ref count right after stuffing the object in the hash because the act of insertion grabs a reference. On checkpoint, this is the reference for the hash. On restore, it would be redundant because it is the first owner (not a file, etc).
Quoted 5 lines
My code takes *another* reference when it attaches to a file, if appropriate. In the case of the adopted socket, only the objhash holds a reference to it, so unless it was joined to a peer, it will be freed when the objhash does its final
obj_ref_drop()at tear-down time.
Yes. On the struct sock. But what will drop the ref on the struct socket? Or has one of your later patches, not yet in ckpt-v17-dev, added that? Or, am I just missing a place where sock_put() will actually sock_release(sk->sk_socket)?
-serge
A message that is not in the archive (sent off-list or not yet received): 20090903144341.GA13182-r/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
SH> Yes. On the struct sock. But what will drop the ref on the SH> struct socket? Or has one of your later patches, not yet in SH> ckpt-v17-dev, added that? Or, am I just missing a place where SH> sock_put() will actually sock_release(sk->sk_socket)?
Hmm, I see what you mean. I can't find any path where sock_put() will release the struct socket. What's weird is that there is a WARN_ON() in af_unix.c:354 that should get tripped if we call sk_free() when we still have a socket. I don't see that, but now I'm not sure why.
Perhaps what we should do is orphan the struct sock before we add it to the hash and then graft it onto a new struct socket before attaching it to a struct file?
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Re: [PATCH 1/3] Make sockets proper objhash objects and use checkpoint_obj() on them (v2)
In reply to a message not in the archiveQuoted 18 lines · Dan Smith
SH> I'm sure I sound like an idiot, but... at restore, a socket will SH> be created for sk now. Is that a problem? I don't see where SH>
sk_free()will cause that sock to be freed, and you are not SH> attaching it do a file whose close would cause it to be SH> released...In the regular socket code, the reference that is taken during allocation is assumed to be for the owning object (the file). So, they don't take another ref when they actually attach it to a file.
The objhash assumes that for all objects on restore, the allocation routine for the object has already incremented the reference count for it as the owner. If you look at
restore_obj()it drops the ref count right after stuffing the object in the hash because the act of insertion grabs a reference. On checkpoint, this is the reference for the hash. On restore, it would be redundant because it is the first owner (not a file, etc).
Quoted 5 lines
My code takes *another* reference when it attaches to a file, if appropriate. In the case of the adopted socket, only the objhash holds a reference to it, so unless it was joined to a peer, it will be freed when the objhash does its final
obj_ref_drop()at tear-down time.
Hmm... one little detail is missing: if this is a dead socket, then the reason we checkpoint it is either:
1) because it is a peer of some other socket: on restart we will 'join' them and bump the reference. This reference will be removed when the other socket is closed - and the (dead) sk will be freed.
2) because it is the source of an skb sent to another socket. The unix socket code will take an extra reference to the (dead) sk in the skb that is placed in the other socket's receive queue. This reference will be dropped when the skb is consumed and the (dead) sk will be freed.
(In both cases, the extra reference keeps the dead sk around after the objhash is cleaned up).
Oren.
PATCH 2/3 Add post-file deferqueue
In reply to a message not in the archiveThis is the deferq bit of Matt's patch, which is needed by the subsequent socket patch.
--- checkpoint/files.c | 28 ++++++++++++++++++++++++++++ include/linux/checkpoint_types.h | 1 + 2 files changed, 29 insertions(+), 0 deletions(-)
checkpoint/files.c +28 -0
@@ -21,6 +21,7 @@#include <linux/syscalls.h> #include <linux/checkpoint.h> #include <linux/checkpoint_hdr.h> +#include <linux/deferqueue.h> #include <net/sock.h>
@@ -289,11 +290,25 @@ static int do_checkpoint_file_table(struct ckpt_ctx *ctx, goto out;ckpt_debug ("nfds %d\n" , nfds); + ctx->files_deferq =deferqueue_create (); +if (!ctx->files_deferq) { + ret = -ENOMEM ; + goto out; + }for (n =0 ; n < nfds; n++) { ret =checkpoint_file_desc (ctx, files, fdtable[n]);if (ret <0 )break ; } +if (!ret) { + ret =deferqueue_run (ctx->files_deferq); +if (ret >0 ) { +pr_warning ("c/r: files deferqueue had %d entries\n" , + ret); + ret =0 ; + } + } +deferqueue_destroy (ctx->files_deferq); out:kfree (fdtable);return ret;
@@ -692,11 +707,24 @@ static struct files_struct *do_restore_file_table(struct ckpt_ctx *ctx)if (ret <0 ) goto out; + ret = -ENOMEM ; + ctx->files_deferq =deferqueue_create (); +if (!ctx->files_deferq) + goto out;for (i =0 ; i < h->fdt_nfds; i++) { ret =restore_file_desc (ctx);if (ret <0 )break ; } +if (!ret) { + ret =deferqueue_run (ctx->files_deferq); +if (ret >0 ) { +pr_warning ("c/r: files deferqueue had %d entries\n" , + ret); + ret =0 ; + } + } +deferqueue_destroy (ctx->files_deferq); out:ckpt_hdr_put (ctx, h);if (!ret) {
include/linux/checkpoint_types.h +1 -0
@@ -48,6 +48,7 @@ struct ckpt_ctx {struct ckpt_obj_hash *obj_hash;/* repository for shared objects */ struct deferqueue_head *deferqueue;/* queue of deferred work */ +struct deferqueue_head *files_deferq;struct path fs_mnt;/* container root (FIXME) */
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1251915760-20118-3-git-send-email-danms-r/[email protected]
Re: [PATCH 2/3] Add post-file deferqueue
Quoted 2 lines · Dan Smith
This is the deferq bit of Matt's patch, which is needed by the subsequent socket patch.
Hmm... I guess you missed my comments to his post - see below:
Quoted 21 lines
Signed-off-by Dan Smith <danms-r/[email protected]>Cc Matt Helsley <matthltc-r/[email protected]>--- checkpoint/files.c | 28 ++++++++++++++++++++++++++++ include/linux/checkpoint_types.h | 1 + 2 files changed, 29 insertions(+), 0 deletions(-)checkpoint/files.c +1
-0@@ -21,6 +21,7 @@#include <linux/syscalls.h> #include <linux/checkpoint.h> #include <linux/checkpoint_hdr.h> +#include <linux/deferqueue.h> #include <net/sock.h>
checkpoint/files.c static int do_checkpoint_file_table(struct ckpt_ctx *ctx, · line 297
Quoted 8 lines
checkpoint/files.c +5
-0@@ -289,11 +290,25 @@ static int do_checkpoint_file_table(struct ckpt_ctx *ctx, goto out;ckpt_debug ("nfds %d\n" , nfds); + ctx->files_deferq =deferqueue_create (); +if (!ctx->files_deferq) { + ret = -ENOMEM ; + goto out; + }
You can create/destroy the deferqueue once, in ckpt_ctx_alloc() and ckpt_ctx_free(), respectively. It will also simplify the logic here.
Quoted 18 lines
checkpoint/files.c +9
-0for (n =0 ; n < nfds; n++) { ret =checkpoint_file_desc (ctx, files, fdtable[n]);if (ret <0 )break ; } +if (!ret) { + ret =deferqueue_run (ctx->files_deferq); +if (ret >0 ) { +pr_warning ("c/r: files deferqueue had %d entries\n" , + ret); + ret =0 ; + } + } +deferqueue_destroy (ctx->files_deferq); out:kfree (fdtable);return ret;
checkpoint/files.c static struct files_struct *do_restore_file_table(struct ckpt_ctx *ctx) · line 713
Quoted 7 lines
checkpoint/files.c +4
-0@@ -692,11 +707,24 @@ static struct files_struct *do_restore_file_table(struct ckpt_ctx *ctx)if (ret <0 ) goto out; + ret = -ENOMEM ; + ctx->files_deferq =deferqueue_create (); +if (!ctx->files_deferq) + goto out;
Ditto for alloc/free of ctx->files_deferq.
Oren.
Quoted 29 lines
checkpoint/files.c +9
-0for (i =0 ; i < h->fdt_nfds; i++) { ret =restore_file_desc (ctx);if (ret <0 )break ; } +if (!ret) { + ret =deferqueue_run (ctx->files_deferq); +if (ret >0 ) { +pr_warning ("c/r: files deferqueue had %d entries\n" , + ret); + ret =0 ; + } + } +deferqueue_destroy (ctx->files_deferq); out:ckpt_hdr_put (ctx, h);if (!ret) {include/linux/checkpoint_types.h +1
-0@@ -48,6 +48,7 @@ struct ckpt_ctx {struct ckpt_obj_hash *obj_hash;/* repository for shared objects */ struct deferqueue_head *deferqueue;/* queue of deferred work */ +struct deferqueue_head *files_deferq;struct path fs_mnt;/* container root (FIXME) */
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 2/3] Add post-file deferqueue
OL> Hmm... I guess you missed my comments to his post - see below:
Yep, I missed them, sorry. I was just including Matt's code almost verbatim here. I'll make those changes.
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
RFC PATCH 3/3 Track socket buffer owners
In reply to a message not in the archiveThis patch is a superset of the previous attempt to store socket buffers with their owners. It defers the writing and reading of the socket buffers until after the end of the file phase to avoid a recursive nose-dive of checkpointing socket owners.
This also moves the join logic to the deferqueue as well, since that too can lead us down a deep hole. The buffer restore logic may perform a join if it decides that the join is inevitable (but not yet performed) and necessary.
Note that I've been staring at this for too long, so I'm sending it as an RFC with hopes that it's not too much of a mess.
--- include/linux/checkpoint.h | 3 - include/linux/checkpoint_hdr.h | 6 + include/linux/net.h | 4 +- include/net/af_unix.h | 4 +- net/checkpoint.c | 131 ++++++++++------ net/unix/checkpoint.c | 339 ++++++++++++++++++++++++++-------------- 6 files changed, 312 insertions(+), 175 deletions(-)
include/linux/checkpoint.h +0 -3
@@ -84,9 +84,6 @@ extern int ckpt_sock_getnames(struct ckpt_ctx *ctx,struct socket *socket, struct sockaddr *loc, unsigned *loc_len, struct sockaddr *rem, unsigned *rem_len); -extern struct ckpt_hdr_socket_queue * -ckpt_sock_read_buffer_hdr (struct ckpt_ctx *ctx, -uint32_t *bufsize);/* ckpt kflags */ #define ckpt_set_ctx_kflag (__ctx, __kflag) \
include/linux/checkpoint_hdr.h +6 -0
@@ -414,6 +414,12 @@ struct ckpt_hdr_socket_queue {__u32 total_bytes; } __attribute__ ((aligned(8))); +struct ckpt_hdr_socket_buffer { +struct ckpt_hdr h; +__s32 src_objref; +__s32 dst_objref; +}; +#define CKPT_UNIX_LINKED 1struct ckpt_hdr_socket_unix {struct ckpt_hdr h;
include/linux/net.h +2 -2
@@ -148,7 +148,7 @@ struct msghdr;struct module ;struct ckpt_ctx ; -struct ckpt_socket ; +struct ckpt_hdr_socket ;struct proto_ops {int family;
@@ -197,7 +197,7 @@ struct proto_ops {int (*checkpoint)(struct ckpt_ctx *ctx,struct socket *sock);int (*restore)(struct ckpt_ctx *ctx,struct socket *sock, -struct ckpt_socket *h); +struct ckpt_hdr_socket *h); };struct net_proto_family {
include/net/af_unix.h +2 -2
@@ -71,10 +71,10 @@ static inline void unix_sysctl_unregister(struct net *net) {}#ifdef CONFIG_CHECKPOINT struct ckpt_ctx ; -struct ckpt_socket ; +struct ckpt_hdr_socket ;extern int unix_checkpoint (struct ckpt_ctx *ctx,struct socket *sock);extern int unix_restore (struct ckpt_ctx *ctx,struct socket *sock, -struct ckpt_socket *h); +struct ckpt_hdr_socket *h);#else #define unix_checkpoint NULL#define unix_restore NULL
net/checkpoint.c +82 -49
228 lines not on this page · raw message
net/unix/checkpoint.c +220 -119
474 lines not on this page · raw message
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1251915760-20118-4-git-send-email-danms-r/[email protected]
Re: [PATCH 3/3] [RFC] Track socket buffer owners
Quoted 10 lines · Dan Smith
This patch is a superset of the previous attempt to store socket buffers with their owners. It defers the writing and reading of the socket buffers until after the end of the file phase to avoid a recursive nose-dive of checkpointing socket owners.
This also moves the join logic to the deferqueue as well, since that too can lead us down a deep hole. The buffer restore logic may perform a join if it decides that the join is inevitable (but not yet performed) and necessary.
Quoted 4 lines
Note that I've been staring at this for too long, so I'm sending it as an RFC with hopes that it's not too much of a mess.
Signed-off-by Dan Smith <danms-r/[email protected]>
Nice, so this solves the chicken-and-egg of socket/peer and the recursion of dgram socket "chains".
IIUC, sockets discovered while writing the receive buffers (being the source of an skb) are checkpointed on the fly. In particular, they are then added to the defer queue.
IOW, they are added to a list that is being traversed (executing the deferred tasks). I think it's OK: deferqueue_add() uses list_add_tail() and deferqueue_run user() uses list_for_each_entry_safe().
This probably deserves a comment ?
[...]
include/linux/checkpoint_hdr.h struct ckpt_hdr_socket_queue { · line 421
Quoted 6 lines
include/linux/checkpoint_hdr.h +5
-0+struct ckpt_hdr_socket_buffer { +struct ckpt_hdr h; +__s32 src_objref; +__s32 dst_objref; +};
Super nit: perhaps use 'sk_objref' for socket, and 'peer_objref' for peer. The latter won't be used in inet sockets. Also, while effectively a no-op, dst_objref is "source" for send buffers :p
[...]
Quoted 11 lines
net/checkpoint.c +11
-0+int sock_deferred_write_buffers (void *data) +{ +struct dq_buffers *dq = (struct dq_buffers *)data; +struct ckpt_ctx *ctx = dq->ctx; +int ret; +int dst_objref; + + dst_objref =ckpt_obj_lookup (ctx, dq->sk,CKPT_OBJ_SOCK ); +if (dst_objref <0 ) { + ckpt_write_err(ctx, + "Unable to get objref of owner socket: %i\n",
net/checkpoint.c static int sock_write_buffers(struct ckpt_ctx *ctx, struct sk_buff_head *queue) · line 193
Quoted 10 lines
net/checkpoint.c +10
-0+ dst_objref); +return dst_objref; + } + + ret =sock_write_buffers (ctx, &dq->sk->sk_receive_queue, dst_objref); +ckpt_debug ("write recv buffers: %i\n" , ret); +if (ret <0 ) +return ret; + + ret =sock_write_buffers (ctx, &dq->sk->sk_write_queue, dst_objref);
^^^^^^^ :p
Quoted 5 lines
net/checkpoint.c +5
-0+ckpt_debug ("write send buffers: %i\n" , ret); + +return ret; +} +
net/checkpoint.c static int sock_write_buffers(struct ckpt_ctx *ctx, struct sk_buff_head *queue) · line 208
Quoted 10 lines
net/checkpoint.c +10
-0+int sock_defer_write_buffers (struct ckpt_ctx *ctx,struct sock *sk) +{ +struct dq_buffers dq; + + dq.ctx = ctx; + dq.sk = sk; + +return deferqueue_add (ctx->files_deferq, &dq,sizeof (dq), + sock_deferred_write_buffers, + sock_deferred_write_buffers);
The dtor should be NULL ?
[...]
net/checkpoint.c static int sock_restore_flags(struct socket *sock, · line 381
Quoted 13 lines
static int sock_restore_flags(struct socket *sock,
net/checkpoint.c +4
-1- struct ckpt_socket *h) + struct ckpt_hdr_socket *h) {int ret;int i;@@ -309,6 +376,9 @@ static int sock_restore_flags(struct socket *sock,return -ENOSYS ; } +if (test_and_clear_bit (SOCK_DEAD , &sk_flags)) +sock_flag (sock->sk,SOCK_DEAD ); +
I think this snippet belongs to the 1st patch.
[...]
net/unix/checkpoint.c line 22
Quoted 10 lines
net/unix/checkpoint.c +10
-0+struct dq_join { +struct ckpt_ctx *ctx; +int src_ref; +int dst_ref; +}; + +struct dq_buffers { +struct ckpt_ctx *ctx; +int sk_ref;/* objref of the socket these buffers belong to */ +};
Another nit: s/ref/objref/ (consistency...)
[...]
net/unix/checkpoint.c static inline int unix_need_cwd(struct sockaddr_un *addr, unsigned long len) · line 76
Quoted 12 lines
net/unix/checkpoint.c +12
-0+static int unix_defer_join (struct ckpt_ctx *ctx, +int src_ref, +int dst_ref) +{ +struct dq_join dq; + + dq.ctx = ctx; + dq.src_ref = src_ref; + dq.dst_ref = dst_ref; + +return deferqueue_add (ctx->files_deferq, &dq,sizeof (dq), + unix_deferred_join, unix_deferred_join);
Here, too, dtor should be NULL ?
[...]
net/unix/checkpoint.c int unix_checkpoint(struct ckpt_ctx *ctx, struct socket *sock) · line 216
Quoted 6 lines
net/unix/checkpoint.c +6
-0+ +/* If we don't have a destination or a peer and we know the +* destination of this skb, then we must need to join with our +* peer +*/ +if (!addrlen && !unix_sk (sk)->peer && (h->dst_objref !=0 )) {
IIUC, h->dst_objref should always be > 0, no ?
[...]
Quoted 19 lines
net/unix/checkpoint.c +19
-0+static int unix_deferred_restore_buffers (void *data) +{ +struct dq_buffers *dq = (struct dq_buffers *)data; +struct ckpt_ctx *ctx = dq->ctx; +struct sock *sk; +struct sockaddr *addr =NULL ; +unsigned int addrlen =0 ; +int ret; + + sk =ckpt_obj_fetch (ctx, dq->sk_ref,CKPT_OBJ_SOCK ); +if (!sk) { +ckpt_debug ("Missing sock ref %i\n" , dq->sk_ref); +return -EINVAL ; + } + +if ((sk->sk_type ==SOCK_DGRAM ) && (unix_sk (sk)->addr !=NULL )) { + addr = (struct sockaddr *)&unix_sk (sk)->addr->name; + addrlen =unix_sk (sk)->addr->len; + }
net/unix/checkpoint.c static int sock_read_buffer_sendmsg(struct ckpt_ctx *ctx, struct sock *sk) · line 341
Quoted 10 lines
net/unix/checkpoint.c +10
-0+ + ret =unix_read_buffers (ctx, addr, addrlen); +ckpt_debug ("read recv buffers: %i\n" , ret); +if (ret <0 ) +return ret; + + ret =unix_read_buffers (ctx, addr, addrlen); +ckpt_debug ("read send buffers: %i\n" , ret); +if (ret !=0 ) + ret = -EINVAL ;/* No send buffers for UNIX sockets */
Change to "if (ret > 0)", otherwise you may mask out a real error.
Quoted 4 lines
net/unix/checkpoint.c +4
-0+ +return ret; +} +
net/unix/checkpoint.c static int sock_read_buffer_sendmsg(struct ckpt_ctx *ctx, struct sock *sk) · line 355
Quoted 10 lines
net/unix/checkpoint.c +10
-0+static int unix_defer_restore_buffers (struct ckpt_ctx *ctx,int sk_ref) +{ +struct dq_buffers dq; + + dq.ctx = ctx; + dq.sk_ref = sk_ref; + +return deferqueue_add (ctx->files_deferq, &dq,sizeof (dq), + unix_deferred_restore_buffers, + unix_deferred_restore_buffers);
... dtor should be NULL.
[...]
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 3/3] [RFC] Track socket buffer owners
OL> Super nit: perhaps use 'sk_objref' for socket, and 'peer_objref' for OL> peer.
Can I use sk_objref and pr_objref? It will annoy me to no end if the underscores don't line up :)
I'll make the other suggested changes, thanks :)
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 87skf4xd9a.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 3/3] [RFC] Track socket buffer owners
Quoted 5 lines · Dan Smith
OL> Super nit: perhaps use '
sk_objref' for socket, and 'peer_objref' for OL> peer.Can I use
sk_objrefandpr_objref? It will annoy me to no end if the underscores don't line up :)
:D
(I coulnd't have said it better...)
Quoted 4 lines
I'll make the other suggested changes, thanks :)