Add support for connected AF_INET sockets
Subsystems (3)
This updated patch set includes fixes for patch 1, fixes and sanity checking for patch 2, and the previous (unchanged) documentation patch as 3. I've not included the TCP time stamp patch and plan to experiment with a few options discussed on IRC. Whatever that is will be a separate patch anyway.
RFC PATCH 2/3 Add c/r support for connected INET sockets (v3)
This patch adds basic support for C/R of open INET sockets. I think that all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
With this patch applied, the following script can be used to demonstrate the functionality:
https://lists.linux-foundation.org/pipermail/containers/2009-October/021239.html
It shows that this enables migration of a sendmail process with open connections from one machine to another without dropping.
We probably need comments from the netdev people about the quality of sanity checking we do on the values in the ckpt_hdr_socket_inet structure on restart.
Note that this still doesn't address lingering sockets yet.
Changes in v3:
- Prevent restart from allowing a bind on a <1024 port unless the user is granted that capability
- Add some sanity checking in the inet_precheck() function to make sure the values read from the checkpoint image are within acceptable ranges
- Check the result of sock_restore_header_info() and fail if needed
Changes in v2:
- Restore saddr, rcv_saddr, daddr, sport, and dport from the sockaddr structure instead of saving them separately
- Fix 'sock' naming in sock_cptrst()
- Don't take the queue lock before skb_queue_tail() since it is done for us
- Allow "listen only" restore behavior if RESTART_SOCK_LISTENONLY flag is specified on sys_restart()
- Pull the implementation of the list of listening sockets back into this patch
- Fix dangling printk
- Add some comments around the parent/child restore logic
--- checkpoint/sys.c | 4 + include/linux/checkpoint.h | 5 +- include/linux/checkpoint_hdr.h | 95 ++++++++++ include/linux/checkpoint_types.h | 2 + net/checkpoint.c | 23 ++-- net/ipv4/checkpoint.c | 349 +++++++++++++++++++++++++++++++++++++- 6 files changed, 463 insertions(+), 15 deletions(-)
checkpoint/sys.c +4 -0
@@ -221,6 +221,8 @@ static void ckpt_ctx_free(struct ckpt_ctx *ctx)kfree (ctx->pids_arr); +sock_listening_list_free (&ctx->listen_sockets); +kfree (ctx); }
@@ -249,6 +251,8 @@ static struct ckpt_ctx *ckpt_ctx_alloc(int fd, unsigned long uflags,spin_lock_init (&ctx->lock);#endif +INIT_LIST_HEAD (&ctx->listen_sockets); + err = -EBADF ; ctx->file =fget (fd);if (!ctx->file)
include/linux/checkpoint.h +4 -1
@@ -19,6 +19,7 @@#define RESTART_TASKSELF 0x1#define RESTART_FROZEN 0x2#define RESTART_GHOST 0x4 +#define RESTART_SOCK_LISTENONLY 0x8#ifdef __KERNEL__#ifdef CONFIG_CHECKPOINT
@@ -48,7 +49,8 @@#define RESTART_USER_FLAGS \ (RESTART_TASKSELF | \ RESTART_FROZEN | \ - RESTART_GHOST) + RESTART_GHOST | \ + RESTART_SOCK_LISTENONLY) extern int walk_task_subtree(struct task_struct *task, int (*func)(struct task_struct *, void *),
@@ -102,6 +104,7 @@ extern int ckpt_sock_getnames(struct ckpt_ctx *ctx, struct sockaddr *rem, unsigned *rem_len);int sock_restore_header_info (struct sk_buff *skb,struct ckpt_hdr_socket_buffer *h); +void sock_listening_list_free (struct list_head *head);/* ckpt kflags */ #define ckpt_set_ctx_kflag (__ctx, __kflag) \
include/linux/checkpoint_hdr.h +95 -0
107 lines not on this page · raw message
include/linux/checkpoint_types.h +2 -0
@@ -65,6 +65,8 @@ struct ckpt_ctx {struct list_head pgarr_list;/* page array to dump VMA contents */ struct list_head pgarr_pool;/* pool of empty page arrays chain */ +struct list_head listen_sockets;/* listening parent sockets */ +/* [multi-process checkpoint] */ struct task_struct **tasks_arr;/* array of all tasks [checkpoint] */ int nr_tasks;/* size of tasks array */
net/checkpoint.c +11 -12
@@ -134,6 +134,7 @@ int sock_restore_header_info(struct sk_buff *skb,static int __sock_write_buffers (struct ckpt_ctx *ctx,struct sk_buff_head *queue, +uint16_t family,int dst_objref) {struct sk_buff *skb;
@@ -142,11 +143,7 @@ static int __sock_write_buffers(struct ckpt_ctx *ctx,struct ckpt_hdr_socket_buffer *h;int ret =0 ; -/* FIXME: This could be a false positive for non-unix -* buffers, so add a type check here in the -* future -*/ -if (UNIXCB (skb).fp) { +if ((family ==AF_UNIX ) &&UNIXCB (skb).fp) {ckpt_write_err (ctx,"TE" ,"af_unix: pass fd" , -EBUSY );return -EBUSY ; }
@@ -186,6 +183,7 @@ static int __sock_write_buffers(struct ckpt_ctx *ctx,static int sock_write_buffers (struct ckpt_ctx *ctx,struct sk_buff_head *queue, +uint16_t family,int dst_objref) {struct ckpt_hdr_socket_queue *h;
@@ -205,7 +203,7 @@ static int sock_write_buffers(struct ckpt_ctx *ctx, h->skb_count = ret; ret =ckpt_write_obj (ctx, (struct ckpt_hdr *) h);if (!ret) - ret =__sock_write_buffers (ctx, &tmpq, dst_objref); + ret =__sock_write_buffers (ctx, &tmpq, family, dst_objref); out:ckpt_hdr_put (ctx, h);
@@ -227,12 +225,14 @@ int sock_deferred_write_buffers(void *data)return dst_objref; } - ret =sock_write_buffers (ctx, &dq->sk->sk_receive_queue, dst_objref); + ret =sock_write_buffers (ctx, &dq->sk->sk_receive_queue, + dq->sk->sk_family, dst_objref);ckpt_debug ("write recv buffers: %i\n" , ret);if (ret <0 )return ret; - ret =sock_write_buffers (ctx, &dq->sk->sk_write_queue, dst_objref); + ret =sock_write_buffers (ctx, &dq->sk->sk_write_queue, + dq->sk->sk_family, dst_objref);ckpt_debug ("write send buffers: %i\n" , ret);return ret;
@@ -757,10 +757,9 @@ struct sock *do_sock_restore(struct ckpt_ctx *ctx) goto err;if ((h->sock_common.family ==AF_INET ) && - (h->sock.state !=TCP_LISTEN )) { -/* Temporary hack to enable restore of TCP_LISTEN sockets -* while forcing anything else to a closed state -*/ + (h->sock.state !=TCP_LISTEN ) && + (ctx->uflags &RESTART_SOCK_LISTENONLY )) { +ckpt_debug ("Forcing open socket closed\n" ); sock->sk->sk_state =TCP_CLOSE ; sock->state =SS_UNCONNECTED ; }
net/ipv4/checkpoint.c +347 -2
394 lines not on this page · raw message
Signature · 1 line
1.6.2.5
Re: [PATCH 2/3] [RFC] Add c/r support for connected INET sockets (v3)
Quoted 39 lines · Dan Smith
This patch adds basic support for C/R of open INET sockets. I think that all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
With this patch applied, the following script can be used to demonstrate the functionality:
https://lists.linux-foundation.org/pipermail/containers/2009-October/021239.html
It shows that this enables migration of a sendmail process with open connections from one machine to another without dropping.
We probably need comments from the netdev people about the quality of sanity checking we do on the values in the
ckpt_hdr_socket_inetstructure on restart.Note that this still doesn't address lingering sockets yet.
Changes in v3:
- Prevent restart from allowing a bind on a <1024 port unless the user is granted that capability
- Add some sanity checking in theinet_precheck()function to make sure the values read from the checkpoint image are within acceptable ranges
- Check the result ofsock_restore_header_info()and fail if neededChanges in v2:
- Restore saddr,rcv_saddr, daddr, sport, and dport from the sockaddr structure instead of saving them separately
- Fix 'sock' naming insock_cptrst()
- Don't take the queue lock beforeskb_queue_tail()since it is done for us
- Allow "listen only" restore behavior ifRESTART_SOCK_LISTENONLYflag is specified onsys_restart()
- Pull the implementation of the list of listening sockets back into this patch
- Fix dangling printk
- Add some comments around the parent/child restore logic
Quoted 5 lines
Cc Oren Laadan <[email protected]>Cc John Dykstra <[email protected]>Signed-off-by Dan Smith <[email protected]>---
This looks good:
I still want to move this to right under the restart-specific fields:
struct list_head listen_sockets;/* listening parent sockets */
Also, I'm looking for a better name for RESTART_SOCK_LISTENONLY (it isn't listen only, udp sockets are preserved...), something that will convey the idea that we drop old connections, perhaps:
RESTART_NET_RESET
RESTART_CONN_RESET
RESTART_DROPCONN
?
Oren.
A message that is not in the archive (sent off-list or not yet received): 1256666008-8231-1-git-send-email-danms-r/[email protected]
PATCH 1/3 Record and restore skb header marks (v2)
Save this information when we checkpoint an skb and provide a mechanism to restore that information on restart. This will be used in the subsequent INET patch.
Changes in v2:
- Add range checks to ensure that the header marks we restore are within the data length of the skb
--- include/linux/checkpoint.h | 2 + include/linux/checkpoint_hdr.h | 7 ++++++ net/checkpoint.c | 45 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 0 deletions(-)
include/linux/checkpoint.h +2 -0
@@ -100,6 +100,8 @@ extern int ckpt_sock_getnames(struct ckpt_ctx *ctx,struct socket *socket, struct sockaddr *loc, unsigned *loc_len, struct sockaddr *rem, unsigned *rem_len); +int sock_restore_header_info (struct sk_buff *skb, +struct ckpt_hdr_socket_buffer *h);/* ckpt kflags */ #define ckpt_set_ctx_kflag (__ctx, __kflag) \
include/linux/checkpoint_hdr.h +7 -0
@@ -542,8 +542,15 @@ struct ckpt_hdr_socket_queue {struct ckpt_hdr_socket_buffer {struct ckpt_hdr h; +__u64 mac_len; +__u64 hdr_len; +__u64 transport_header; +__u64 network_header; +__u64 mac_header;__s32 sk_objref;__s32 pr_objref; +__u16 protocol; +__u8 cb[48 ]; };#define CKPT_UNIX_LINKED 1
net/checkpoint.c +45 -0
57 lines not on this page · raw message
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1256666008-8231-2-git-send-email-danms-r/[email protected]
Re: [PATCH 1/3] Record and restore skb header marks (v2)
Quoted 9 lines · Dan Smith
Save this information when we checkpoint an skb and provide a mechanism to restore that information on restart. This will be used in the subsequent INET patch.
Signed-off-by Dan Smith <danms-r/[email protected]>Changes in v2:
- Add range checks to ensure that the header marks we restore are within the data length of the skb
[...]
net/checkpoint.c static int sock_copy_buffers(struct sk_buff_head *from, · line 118
Quoted 8 lines
net/checkpoint.c +8
-0+int sock_restore_header_info (struct sk_buff *skb, +struct ckpt_hdr_socket_buffer *h) +{ +if ((h->transport_header >= skb->len) || + (h->network_header >= skb->len) || + (h->mac_header >= skb->len) || + (h->mac_len >= skb->len) || + (h->hdr_len >= skb->len)) {
I wonder if the sanity test for mac_len and hdr_len are sufficient, or whether a more constrained test is required.
For example, a quick grep shows that there are several places where:
skb->mac_len = skb->network_header - skb->mac_header;
I think there is some code that relies on it without validating that the value makes sense.
net/checkpoint.c static int sock_copy_buffers(struct sk_buff_head *from, · line 130
Quoted 12 lines
net/checkpoint.c +12
-0+ckpt_debug ("skb header positions not within data length\n" ); +return -EINVAL ; + } + + skb->mac_len = h->mac_len; + skb->hdr_len = h->hdr_len; + +skb_set_transport_header (skb, h->transport_header); +skb_set_network_header (skb, h->network_header); +skb_set_mac_header (skb, h->mac_header); + +memcpy (skb->cb, h->cb,sizeof (skb->cb));
The skb->cb holds can be used by any layer to put private variables.
Can the user mangle the data in there to create a disaster of some sort ?
If the answer is "it's possible", and because this is per protocol data, I suggest to add a per-protocol callback to sanitize the data in this control buffer.
To not block this patchset infinitely, I guess you can put the details of the sanity check in a separate patch(set). But I prefer that the current set will at least mention and provision for such a mechanism.
Quoted 8 lines
net/checkpoint.c +4
-0+ +return 0 ; +} +static int __sock_write_buffers (struct ckpt_ctx *ctx,struct sk_buff_head *queue,int dst_objref)
net/checkpoint.c static int __sock_write_buffers(struct ckpt_ctx *ctx, · line 173
Quoted 7 lines
net/checkpoint.c +1
-0@@ -123,6 +167,7 @@ static int __sock_write_buffers(struct ckpt_ctx *ctx, goto end; h->sk_objref = ret; h->pr_objref = dst_objref; +sock_record_header_info (skb, h); ret =ckpt_write_obj (ctx, (struct ckpt_hdr *) h);if (ret <0 )
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/3] Record and restore skb header marks (v2)
Eesh, I just realized I never replied to this mail. Sorry about that.
OL> I wonder if the sanity test for mac_len and hdr_len are OL> sufficient, or whether a more constrained test is required.
Yep, I have it changed now, along with some of the other checks.
OL> The skb->cb holds can be used by any layer to put private OL> variables.
OL> Can the user mangle the data in there to create a disaster of some OL> sort ?
OL> If the answer is "it's possible", and because this is per protocol OL> data, I suggest to add a per-protocol callback to sanitize the OL> data in this control buffer.
Okay, then my answer is "it could be possible later". Right now, I don't think there's anything in there that could be used to do more harm than any of the other things we restore for TCP. We don't restore it for UNIX sockets.
OL> To not block this patchset infinitely, I guess you can put the OL> details of the sanity check in a separate patch(set). But I prefer OL> that the current set will at least mention and provision for such OL> a mechanism.
Indeed. I've added a lengthy comment to be included in the next posting to cover it for now.
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
PATCH 3/3 Add some content to the readme.txt for socket c/r
In reply to a message not in the archive--- Documentation/checkpoint/readme.txt | 21 +++++++++++++++++++++ 1 files changed, 21 insertions(+), 0 deletions(-)
Documentation/checkpoint/readme.txt +21 -0
@@ -334,6 +334,27 @@ we will be forced to more carefully review each of those features. However, this can be controlled with a sysctl-variable. +Sockets +======= + +For AF_UNIX sockets, both endpoints must be within the checkpointed +task set to maintain a connected state after restart. UNIX sockets +that are in the process of passing a descriptor will cause the +checkpoint to fail with -EBUSY indicating a transient state that +cannot be checkpointed. Listening sockets with an unaccepted peer +will also cause an -EBUSY result. + +AF_INET sockets with endpoints outside the checkpointed task set may +remain open if care is taken to avoid TCP timeouts and resets. +Careful use of a virtual IP address can help avoid emission of an RST +to the non-checkpointed endpoint. If desired, the +RESTART_SOCK_LISTENONLY flag may be passed to the restart syscall +which will cause all connected AF_INET sockets to be closed during the +restore process. Listening sockets will still be restored to their +original state, which makes this mode a candidate for something like +an HTTP server. + + Kernel interfaces =================
Signature · 1 line
1.6.2.5