c/r: Support open INET connections
Subsystems (2)
This set adds support for open INET connection migration. To be tested with the recently-posted demonstration script.
A message that is not in the archive (sent off-list or not yet received): 1254932945-12578-1-git-send-email-danms-r/[email protected]
PATCH 1/2 Add socket list helper functions
The INET patch needs to maintain a list of listening sockets in order to support post-restart socket hashing. The ckpt_ctx_free() function thus needs to clean up that list after processing the deferqueue, and needs to know something about what is on that list. Instead of calling into a single cleanup function in the checkpoint code, I decided to just make the list itself a little more abstract and separate it here.
--- include/linux/checkpoint.h | 5 +++++ net/checkpoint.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 0 deletions(-)
include/linux/checkpoint.h +5 -0
@@ -100,6 +100,11 @@ extern int ckpt_sock_getnames(struct ckpt_ctx *ctx,struct socket *socket, struct sockaddr *loc, unsigned *loc_len, struct sockaddr *rem, unsigned *rem_len); +int sock_list_add (struct list_head *list,struct sock *sk); +typedef int (*sock_compare_fn )(struct sock *,struct sock *); +struct sock *sock_list_find (struct list_head *list,struct sock *sk, +sock_compare_fn cmp); +void sock_list_free (struct list_head *list);/* ckpt kflags */ #define ckpt_set_ctx_kflag (__ctx, __kflag) \
net/checkpoint.c +42 -0
45 lines not on this page · raw message
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1254932945-12578-2-git-send-email-danms-r/[email protected]
Re: [PATCH 1/2] Add socket list helper functions
Quoted 6 lines · Dan Smith
The INET patch needs to maintain a list of listening sockets in order to support post-restart socket hashing. The
ckpt_ctx_free()function thus needs to clean up that list after processing the deferqueue, and needs to know something about what is on that list. Instead of calling into a single cleanup function in the checkpoint code, I decided to just make the list itself a little more abstract and separate it here.
IMHO this is over-engineered. This way, you place net-related knowledge in ckpt_ctx_free(), instead of isolating that sort of smarts under net/ subdir. I'd rather see the clean-up logic near the "litter" logic, so to speak.
Or, make it entirely generic, use container_of etc and put in, for instance, in include/linux/list2.h. Anyway, I don't think it belongs in checkpoint.h.
Thanks,
Oren.
Quoted 73 lines
Signed-off-by Dan Smith <danms-r/[email protected]>--- include/linux/checkpoint.h | 5 +++++ net/checkpoint.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 0 deletions(-)include/linux/checkpoint.h +5
-0@@ -100,6 +100,11 @@ extern int ckpt_sock_getnames(struct ckpt_ctx *ctx,struct socket *socket, struct sockaddr *loc, unsigned *loc_len, struct sockaddr *rem, unsigned *rem_len); +int sock_list_add (struct list_head *list,struct sock *sk); +typedef int (*sock_compare_fn )(struct sock *,struct sock *); +struct sock *sock_list_find (struct list_head *list,struct sock *sk, +sock_compare_fn cmp); +void sock_list_free (struct list_head *list);/* ckpt kflags */ #define ckpt_set_ctx_kflag (__ctx, __kflag) \net/checkpoint.c +42
-0@@ -758,3 +758,45 @@ struct file *sock_file_restore(struct ckpt_ctx *ctx, struct ckpt_hdr_file *ptr)return file; } + +struct ckpt_sock_list_item { +struct sock *sk; +struct list_head list; +}; + +int sock_list_add (struct list_head *list,struct sock *sk) +{ +struct ckpt_sock_list_item *item; + + item =kmalloc (sizeof (*item),GFP_KERNEL ); +if (!item) +return -ENOMEM ; + + item->sk = sk; +list_add (&item->list, list); + +return 0 ; +} + +void sock_list_free (struct list_head *list) +{ +struct ckpt_sock_list_item *item, *tmp; + +list_for_each_entry_safe (item, tmp, list, list) { +list_del (&item->list); +kfree (item); + } +} + +struct sock *sock_list_find (struct list_head *list,struct sock *sk, +sock_compare_fn cmp) +{ +struct ckpt_sock_list_item *item; + +list_for_each_entry (item, list, list) { +if (cmp (sk, item->sk)) +return item->sk; + } + +return NULL ; +}
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/2] Add socket list helper functions
OL> IMHO this is over-engineered. This way, you place net-related OL> knowledge in ckpt_ctx_free(), instead of isolating that sort of OL> smarts under net/ subdir. I'd rather see the clean-up logic near OL> the "litter" logic, so to speak.
Well, I still need to clean up the list after the deferqueue has run, so something has to go into ckpt_ctx_free() to do that. I can make it a specific call like ckpt_sock_cleanup_listening() or something, but that will still need to go in checkpoint.h so I can call it from ctx_free()...
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 87d44re36u.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 1/2] Add socket list helper functions
Quoted 10 lines · Dan Smith
OL> IMHO this is over-engineered. This way, you place net-related OL> knowledge in
ckpt_ctx_free(), instead of isolating that sort of OL> smarts undernet/subdir. I'd rather see the clean-up logic near OL> the "litter" logic, so to speak.Well, I still need to clean up the list after the deferqueue has run, so something has to go into
ckpt_ctx_free()to do that. I can make it a specific call likeckpt_sock_cleanup_listening()or something, but that will still need to go incheckpoint.hso I can call it fromctx_free()...
Heh, but that sounds like over-engineering :)
-serge
RFC PATCH 2/2 Add c/r support for connected INET sockets
In reply to a message not in the archiveThis patch adds basic support for C/R of open INET sockets. I think that all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
With this patch applied, the following script can be used to demonstrate the functionality:
https://lists.linux-foundation.org/pipermail/containers/2009-October/021239.html
It shows that this enables migration of a sendmail process with open connections from one machine to another without dropping.
Now that listening socket support is in the c/r tree, I think it is a good time to start fielding comments and suggestions on the connected part, as I think lots of folks have input on how to make it better, safer, etc.
--- checkpoint/sys.c | 4 + include/linux/checkpoint_hdr.h | 97 +++++++++++++++++++ include/linux/checkpoint_types.h | 2 + net/checkpoint.c | 25 ++---- net/ipv4/checkpoint.c | 192 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 303 insertions(+), 17 deletions(-)
checkpoint/sys.c +4 -0
@@ -221,6 +221,8 @@ static void ckpt_ctx_free(struct ckpt_ctx *ctx)kfree (ctx->pids_arr); +sock_list_free (&ctx->listen_sockets); +kfree (ctx); }
@@ -249,6 +251,8 @@ static struct ckpt_ctx *ckpt_ctx_alloc(int fd, unsigned long uflags,spin_lock_init (&ctx->lock);#endif +INIT_LIST_HEAD (&ctx->listen_sockets); + err = -EBADF ; ctx->file =fget (fd);if (!ctx->file)
include/linux/checkpoint_hdr.h +97 -0
109 lines not on this page · raw message
include/linux/checkpoint_types.h +2 -0
@@ -65,6 +65,8 @@ struct ckpt_ctx {struct list_head pgarr_list;/* page array to dump VMA contents */ struct list_head pgarr_pool;/* pool of empty page arrays chain */ +struct list_head listen_sockets;/* listening parent sockets */ +/* [multi-process checkpoint] */ struct task_struct **tasks_arr;/* array of all tasks [checkpoint] */ int nr_tasks;/* size of tasks array */
net/checkpoint.c +8 -17
@@ -90,6 +90,7 @@ static int sock_copy_buffers(struct sk_buff_head *from,static int __sock_write_buffers (struct ckpt_ctx *ctx,struct sk_buff_head *queue, +uint16_t family,int dst_objref) {struct sk_buff *skb;
@@ -98,11 +99,7 @@ static int __sock_write_buffers(struct ckpt_ctx *ctx,struct ckpt_hdr_socket_buffer *h;int ret =0 ; -/* FIXME: This could be a false positive for non-unix -* buffers, so add a type check here in the -* future -*/ -if (UNIXCB (skb).fp) { +if ((family ==AF_UNIX ) &&UNIXCB (skb).fp) {ckpt_write_err (ctx,"TE" ,"af_unix: pass fd" , -EBUSY );return -EBUSY ; }
@@ -141,6 +138,7 @@ static int __sock_write_buffers(struct ckpt_ctx *ctx,static int sock_write_buffers (struct ckpt_ctx *ctx,struct sk_buff_head *queue, +uint16_t family,int dst_objref) {struct ckpt_hdr_socket_queue *h;
@@ -160,7 +158,7 @@ static int sock_write_buffers(struct ckpt_ctx *ctx, h->skb_count = ret; ret =ckpt_write_obj (ctx, (struct ckpt_hdr *) h);if (!ret) - ret =__sock_write_buffers (ctx, &tmpq, dst_objref); + ret =__sock_write_buffers (ctx, &tmpq, family, dst_objref); out:ckpt_hdr_put (ctx, h);
@@ -182,12 +180,14 @@ int sock_deferred_write_buffers(void *data)return dst_objref; } - ret =sock_write_buffers (ctx, &dq->sk->sk_receive_queue, dst_objref); + ret =sock_write_buffers (ctx, &dq->sk->sk_receive_queue, + dq->sk->sk_family, dst_objref);ckpt_debug ("write recv buffers: %i\n" , ret);if (ret <0 )return ret; - ret =sock_write_buffers (ctx, &dq->sk->sk_write_queue, dst_objref); + ret =sock_write_buffers (ctx, &dq->sk->sk_write_queue, + dq->sk->sk_family, dst_objref);ckpt_debug ("write send buffers: %i\n" , ret);return ret;
@@ -710,15 +710,6 @@ struct sock *do_sock_restore(struct ckpt_ctx *ctx)if (ret <0 ) goto err; -if ((h->sock_common.family ==AF_INET ) && - (h->sock.state !=TCP_LISTEN )) { -/* Temporary hack to enable restore of TCP_LISTEN sockets -* while forcing anything else to a closed state -*/ - sock->sk->sk_state =TCP_CLOSE ; - sock->state =SS_UNCONNECTED ; - } -ckpt_hdr_put (ctx, h);return sock->sk; err:
net/ipv4/checkpoint.c +192 -0
218 lines not on this page · raw message
Signature · 1 line
1.6.2.5
A message that is not in the archive (sent off-list or not yet received): 1254932945-12578-3-git-send-email-danms-r/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 5 lines · Dan Smith
This patch adds basic support for C/R of open INET sockets. I think that all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
Quoted 7 lines
With this patch applied, the following script can be used to demonstrate the functionality:
https://lists.linux-foundation.org/pipermail/containers/2009-October/021239.html
It shows that this enables migration of a sendmail process with open connections from one machine to another without dropping.
neato
Quoted 4 lines
Now that listening socket support is in the c/r tree, I think it is a good time to start fielding comments and suggestions on the connected part, as I think lots of folks have input on how to make it better, safer, etc.
One thing:
Quoted 5 lines
net/ipv4/checkpoint.c +5
-0+static int sock_inet_cptrst (struct ckpt_ctx *ctx, +struct sock *sock, +struct ckpt_hdr_socket_inet *hh, +int op) +{
net/ipv4/checkpoint.c struct dq_buffers { · line 161
Quoted 10 lines
net/ipv4/checkpoint.c +10
-0+struct inet_sock *sk =inet_sk (sock); +struct inet_connection_sock *icsk =inet_csk (sock); +int ret; + +CKPT_COPY (op, hh->daddr, sk->daddr); +CKPT_COPY (op, hh->rcv_saddr, sk->rcv_saddr); +CKPT_COPY (op, hh->dport, sk->dport); +CKPT_COPY (op, hh->num, sk->num); +CKPT_COPY (op, hh->saddr, sk->saddr); +CKPT_COPY (op, hh->sport, sk->sport);
This becomes an easy way around CAP_NET_BIND_SERVICE right? Or will that be caught by something already done in your listen patch after this step?
thanks,
-serge
A message that is not in the archive (sent off-list or not yet received): 20091007171907.GA20572-r/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 6 lines
net/ipv4/checkpoint.c struct dq_buffers { · line 161
Quoted 6 lines
net/ipv4/checkpoint.c +6
-0+CKPT_COPY (op, hh->daddr, sk->daddr); +CKPT_COPY (op, hh->rcv_saddr, sk->rcv_saddr); +CKPT_COPY (op, hh->dport, sk->dport); +CKPT_COPY (op, hh->num, sk->num); +CKPT_COPY (op, hh->saddr, sk->saddr); +CKPT_COPY (op, hh->sport, sk->sport);
SH> This becomes an easy way around CAP_NET_BIND_SERVICE right? Or SH> will that be caught by something already done in your listen patch SH> after this step?
Actually, yeah, you're right. I was going to say that we'd catch it because we also do a bind(), but there's no guarantee that the sockaddr_in we use for bind() is the same as this :D
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
In reply to Dan SmithQuoted 5 lines · Dan Smith
This patch adds basic support for C/R of open INET sockets. I think that
all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
I think this patch breaks code that was already in do_sock_restore():
struct sock *do_sock_restore(struct ckpt_ctx *ctx)
{
struct ckpt_hdr_socket *h;
struct socket *sock;
int ret;
h = ckpt_read_obj_type(ctx, sizeof(*h), CKPT_HDR_SOCKET);
if (IS_ERR(h))
return ERR_PTR(PTR_ERR(h));
/* silently clear flags, e.g. SOCK_NONBLOCK or SOCK_CLOEXEC */
h->sock.type &= SOCK_TYPE_MASK;
ret = sock_create(h->sock_common.family, h->sock.type, 0, &sock);
if (ret < 0)
goto err;You're passing 0 as the protocol value to sock_create(). This ultimately gets passed to the address family's create() function.
inet_create() (and its IPv6 companion) use that protocol value as the key when they search for the proper inet_protosw, which in turn gets mapped to the struct proto and passed to sk_prot_alloc().
In address families INET and AF_INET6, the struct sock is different sizes for different protocols. This is implemented by the struct proto specifying which cache the struct sock comes from.
So by passing in 0 all the time to sock_create(), you're getting a struct sock that may not be the right size. Memory corruption and madness follow.
-- John
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
JD> You're passing 0 as the protocol value to sock_create(). This JD> ultimately gets passed to the address family's create() function.
The fix is to pass in the previous version of sk_protocol instead of zero, right? It looks like inet_create() has enough intelligence to weed out invalid values for us too...
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 87ab01gag7.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 5 lines · Dan Smith
JD> You're passing 0 as the protocol value to
sock_create(). This JD> ultimately gets passed to the address family'screate()function.The fix is to pass in the previous version of
sk_protocolinstead of zero, right?
Yep. It sort of messes up your separation between layers, though.
-- John
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
JD> Yep. It sort of messes up your separation between layers, though.
In what way? It's just this:
net/checkpoint.c +2 -1
@@ -687,7 +687,8 @@ struct sock *do_sock_restore(struct ckpt_ctx *ctx)/* silently clear flags, e.g. SOCK_NONBLOCK or SOCK_CLOEXEC */ h->sock.type &=SOCK_TYPE_MASK ; - ret =sock_create (h->sock_common.family, h->sock.type,0 , &sock); + ret =sock_create (h->sock_common.family, h->sock.type, + h->sock.protocol, &sock);if (ret <0 ) goto err;
Which seems like a why-didn't-I-do-that-in-the-first-place sort of
thing... Am I missing something?
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
A message that is not in the archive (sent off-list or not yet received): 8763apg57w.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 3 lines · Dan Smith
JD> Yep. It sort of messes up your separation between layers, though.
In what way? It's just this:
I just meant that you're peeking into the checkpointed sk data before you create the new sk. In so much of your code, you manage to restore structs as a whole, without peeking first.
Quoted 2 lines
Which seems like a why-didn't-I-do-that-in-the-first-place sort of
thing... Am I missing something?
Not anything that I see.
-- John
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
JD> I just meant that you're peeking into the checkpointed sk data JD> before you create the new sk. In so much of your code, you manage JD> to restore structs as a whole, without peeking first.
Ah, okay, gotcha.
Thanks!
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
In reply to Dan SmithQuoted 10 lines · Dan Smith
This patch adds basic support for C/R of open INET sockets. I think that all the important bits of the TCP and ICSK socket structures is saved, but I think there is still some additional IPv6 stuff that needs to be handled.
With this patch applied, the following script can be used to demonstrate the functionality:
https://lists.linux-foundation.org/pipermail/containers/2009-October/021239.html
Quoted 7 lines
It shows that this enables migration of a sendmail process with open connections from one machine to another without dropping.
Now that listening socket support is in the c/r tree, I think it is a good time to start fielding comments and suggestions on the connected part, as I think lots of folks have input on how to make it better, safer, etc.
Nice !
Quoted 6 lines
Cc Oren Laadan <[email protected]>Cc John Dykstra <[email protected]>Signed-off-by Dan Smith <[email protected]>---
A few comments/questions before the review:
* Did you test this with UDP too ?
* What happens if the the clock on the target machine differs from the
clock on the origin machine ? (TCP timestamps)
* How confident are we that "bad" input in one or more fields, that you don't currently sanitize, cannot create "bad" behavior ? (bad can be kernel crash, unauthorized behavior, DoS etc)
* How much does TCP rely on the validity of the info in the protocol control block, and what sorts of bads can happen if it isn't ? Would TCP be still happy if the URG point is bogus, would it allow the user to sent packets otherwise disallowed (to that user?), or maybe it could crash the kernel ?
* In other words, how much input-sanity-logic is missing (if any) ?
* Can you please document (brief description) how the restart logic works (listening parent socket etc) ?
* Do you intend to checkpoint (and collect) lingering sockets, that is they are closed by the application so not references by any task, but still sending data from their buffers ?
* I'd like to also preserve the "older" behavior - so the user can choose to restart and reset all previous connections, keep listening sockets (e.g. RESTART_DISCONNET).
Also, a couple nits inside below.
Quoted 7 lines
checkpoint/sys.c | 4 + include/linux/checkpoint_hdr.h | 97 +++++++++++++++++++ include/linux/checkpoint_types.h | 2 + net/checkpoint.c | 25 ++---- net/ipv4/checkpoint.c | 192 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 303 insertions(+), 17 deletions(-)
[...]
include/linux/checkpoint_hdr.h struct ckpt_hdr_socket_unix { · line 487
Quoted 13 lines
include/linux/checkpoint_hdr.h +9
-0@@ -475,6 +476,102 @@ struct ckpt_hdr_socket_unix {struct ckpt_hdr_socket_inet {struct ckpt_hdr h; +__u32 daddr; +__u32 rcv_saddr; +__u32 saddr; +__u16 dport; +__u16 num; +__u16 sport; +__s16 uc_ttl; +__u16 cmsg_flags; +
Alignment ?
Quoted 12 lines
include/linux/checkpoint_hdr.h +12
-0+struct { +__u64 timeout; +__u32 ato; +__u32 lrcvtime; +__u16 last_seg_size; +__u16 rcv_mss; +__u8 pending; +__u8 quick; +__u8 pingpong; +__u8 blocked; + } icsk_ack __attribute__ ((aligned(8))); +
[...]
Quoted 6 lines
net/checkpoint.c +0
-2@@ -710,15 +710,6 @@ struct sock *do_sock_restore(struct ckpt_ctx *ctx)if (ret <0 ) goto err; -if ((h->sock_common.family ==AF_INET ) && - (h->sock.state !=TCP_LISTEN )) {
net/checkpoint.c struct sock *do_sock_restore(struct ckpt_ctx *ctx) · line 715
Quoted 10 lines
net/checkpoint.c +0
-7-/* Temporary hack to enable restore of TCP_LISTEN sockets -* while forcing anything else to a closed state -*/ - sock->sk->sk_state =TCP_CLOSE ; - sock->state =SS_UNCONNECTED ; - } -ckpt_hdr_put (ctx, h);return sock->sk; err:
How about leaving this inside if the user explicitly requests so, e.g. using a RESTART_DISCONNECT or something ?
Quoted 39 lines
net/ipv4/checkpoint.c +24
-0@@ -17,6 +17,7 @@#include <linux/deferqueue.h> #include <net/tcp_states.h> #include <net/tcp.h> +#include <net/ipv6.h> struct dq_sock {struct ckpt_ctx *ctx;@@ -28,6 +29,176 @@ struct dq_buffers {struct sock *sk; }; +static int sock_is_parent (struct sock *sk,struct sock *parent) +{ +return inet_sk (sk)->sport ==inet_sk (parent)->sport; +} + +static struct sock *sock_get_parent (struct ckpt_ctx *ctx,struct sock *sk) +{ +return sock_list_find (&ctx->listen_sockets, sk, sock_is_parent); +} + +static int sock_hash_parent (void *data) +{ +struct dq_sock *dq = (struct dq_sock *)data; +struct sock *parent; + +printk ("Doing post-restart hash\n" ); + + dq->sk->sk_prot->hash (dq->sk); + + parent =sock_get_parent (dq->ctx, dq->sk); +if (parent) { +inet_sk (dq->sk)->num =ntohs (inet_sk (dq->sk)->sport); +local_bh_disable ();
net/ipv4/checkpoint.c struct dq_buffers { · line 64
Quoted 10 lines
net/ipv4/checkpoint.c +10
-0+__inet_inherit_port (parent, dq->sk); +local_bh_enable (); + }else { +inet_sk (dq->sk)->num =0 ; +inet_hash_connect (&tcp_death_row, dq->sk); +inet_sk (dq->sk)->num =ntohs (inet_sk (dq->sk)->sport); + } + +return 0 ; +}
I wonder if a user can use this to convince TCP to send some nasty packets to some arbitrary destination, with specific seq-number or what not ?
Quoted 7 lines
net/ipv4/checkpoint.c +7
-0+ +static int sock_defer_hash (struct ckpt_ctx *ctx,struct sock *sock) +{ +struct dq_sock dq; + + dq.sk = sock; + dq.ctx = ctx;
net/ipv4/checkpoint.c struct dq_buffers { · line 81
Quoted 10 lines
net/ipv4/checkpoint.c +10
-0+ +return deferqueue_add (ctx->deferqueue, &dq,sizeof (dq), + sock_hash_parent,NULL ); +} + +static int sock_inet_tcp_cptrst (struct ckpt_ctx *ctx, +struct tcp_sock *sk, +struct ckpt_hdr_socket_inet *hh, +int op) +{
Which of the following list is safe to restore *as is*, and which requires sanitation tests ?
Quoted 110 lines
net/ipv4/checkpoint.c +110
-0+CKPT_COPY (op, hh->tcp.rcv_nxt, sk->rcv_nxt); +CKPT_COPY (op, hh->tcp.copied_seq, sk->copied_seq); +CKPT_COPY (op, hh->tcp.rcv_wup, sk->rcv_wup); +CKPT_COPY (op, hh->tcp.snd_nxt, sk->snd_nxt); +CKPT_COPY (op, hh->tcp.snd_una, sk->snd_una); +CKPT_COPY (op, hh->tcp.snd_sml, sk->snd_sml); +CKPT_COPY (op, hh->tcp.rcv_tstamp, sk->rcv_tstamp); +CKPT_COPY (op, hh->tcp.lsndtime, sk->lsndtime); + +CKPT_COPY (op, hh->tcp.snd_wl1, sk->snd_wl1); +CKPT_COPY (op, hh->tcp.snd_wnd, sk->snd_wnd); +CKPT_COPY (op, hh->tcp.max_window, sk->max_window); +CKPT_COPY (op, hh->tcp.mss_cache, sk->mss_cache); +CKPT_COPY (op, hh->tcp.window_clamp, sk->window_clamp); +CKPT_COPY (op, hh->tcp.rcv_ssthresh, sk->rcv_ssthresh); +CKPT_COPY (op, hh->tcp.frto_highmark, sk->frto_highmark); +CKPT_COPY (op, hh->tcp.advmss, sk->advmss); +CKPT_COPY (op, hh->tcp.frto_counter, sk->frto_counter); +CKPT_COPY (op, hh->tcp.nonagle, sk->nonagle); + +CKPT_COPY (op, hh->tcp.srtt, sk->srtt); +CKPT_COPY (op, hh->tcp.mdev, sk->mdev); +CKPT_COPY (op, hh->tcp.mdev_max, sk->mdev_max); +CKPT_COPY (op, hh->tcp.rttvar, sk->rttvar); +CKPT_COPY (op, hh->tcp.rtt_seq, sk->rtt_seq); + +CKPT_COPY (op, hh->tcp.packets_out, sk->packets_out); +CKPT_COPY (op, hh->tcp.retrans_out, sk->retrans_out); + +CKPT_COPY (op, hh->tcp.urg_data, sk->urg_data); +CKPT_COPY (op, hh->tcp.ecn_flags, sk->ecn_flags); +CKPT_COPY (op, hh->tcp.reordering, sk->reordering); +CKPT_COPY (op, hh->tcp.snd_up, sk->snd_up); + +CKPT_COPY (op, hh->tcp.keepalive_probes, sk->keepalive_probes); + +CKPT_COPY (op, hh->tcp.rcv_wnd, sk->rcv_wnd); +CKPT_COPY (op, hh->tcp.write_seq, sk->write_seq); +CKPT_COPY (op, hh->tcp.pushed_seq, sk->pushed_seq); +CKPT_COPY (op, hh->tcp.lost_out, sk->lost_out); +CKPT_COPY (op, hh->tcp.sacked_out, sk->sacked_out); +CKPT_COPY (op, hh->tcp.fackets_out, sk->fackets_out); +CKPT_COPY (op, hh->tcp.tso_deferred, sk->tso_deferred); +CKPT_COPY (op, hh->tcp.bytes_acked, sk->bytes_acked); + +CKPT_COPY (op, hh->tcp.lost_cnt_hint, sk->lost_cnt_hint); +CKPT_COPY (op, hh->tcp.retransmit_high, sk->retransmit_high); + +CKPT_COPY (op, hh->tcp.lost_retrans_low, sk->lost_retrans_low); + +CKPT_COPY (op, hh->tcp.prior_ssthresh, sk->prior_ssthresh); +CKPT_COPY (op, hh->tcp.high_seq, sk->high_seq); + +CKPT_COPY (op, hh->tcp.retrans_stamp, sk->retrans_stamp); +CKPT_COPY (op, hh->tcp.undo_marker, sk->undo_marker); +CKPT_COPY (op, hh->tcp.undo_retrans, sk->undo_retrans); +CKPT_COPY (op, hh->tcp.total_retrans, sk->total_retrans); + +CKPT_COPY (op, hh->tcp.urg_seq, sk->urg_seq); +CKPT_COPY (op, hh->tcp.keepalive_time, sk->keepalive_time); +CKPT_COPY (op, hh->tcp.keepalive_intvl, sk->keepalive_intvl); + +return 0 ; +} + +static int sock_inet_cptrst (struct ckpt_ctx *ctx, +struct sock *sock, +struct ckpt_hdr_socket_inet *hh, +int op) +{ +struct inet_sock *sk =inet_sk (sock); +struct inet_connection_sock *icsk =inet_csk (sock); +int ret; + +CKPT_COPY (op, hh->daddr, sk->daddr); +CKPT_COPY (op, hh->rcv_saddr, sk->rcv_saddr); +CKPT_COPY (op, hh->dport, sk->dport); +CKPT_COPY (op, hh->num, sk->num); +CKPT_COPY (op, hh->saddr, sk->saddr); +CKPT_COPY (op, hh->sport, sk->sport); +CKPT_COPY (op, hh->uc_ttl, sk->uc_ttl); +CKPT_COPY (op, hh->cmsg_flags, sk->cmsg_flags); + +CKPT_COPY (op, hh->icsk_ack.pending, icsk->icsk_ack.pending); +CKPT_COPY (op, hh->icsk_ack.quick, icsk->icsk_ack.quick); +CKPT_COPY (op, hh->icsk_ack.pingpong, icsk->icsk_ack.pingpong); +CKPT_COPY (op, hh->icsk_ack.blocked, icsk->icsk_ack.blocked); +CKPT_COPY (op, hh->icsk_ack.ato, icsk->icsk_ack.ato); +CKPT_COPY (op, hh->icsk_ack.timeout, icsk->icsk_ack.timeout); +CKPT_COPY (op, hh->icsk_ack.lrcvtime, icsk->icsk_ack.lrcvtime); +CKPT_COPY (op, + hh->icsk_ack.last_seg_size, icsk->icsk_ack.last_seg_size); +CKPT_COPY (op, hh->icsk_ack.rcv_mss, icsk->icsk_ack.rcv_mss); + +if (sock->sk_protocol ==IPPROTO_TCP ) + ret =sock_inet_tcp_cptrst (ctx,tcp_sk (sock), hh, op); +else if (sock->sk_protocol ==IPPROTO_UDP ) + ret =0 ; +else { +ckpt_write_err (ctx,"T" ,"unknown socket protocol %d" , + sock->sk_protocol); + ret = -EINVAL ; + } + +if (sock->sk_family ==AF_INET6 ) { +struct ipv6_pinfo *inet6 =inet6_sk (sock); +if (op ==CKPT_CPT ) { +ipv6_addr_copy (&hh->inet6.saddr, &inet6->saddr); +ipv6_addr_copy (&hh->inet6.rcv_saddr, &inet6->rcv_saddr); +ipv6_addr_copy (&hh->inet6.daddr, &inet6->daddr);
net/ipv4/checkpoint.c struct dq_buffers { · line 201
Quoted 10 lines
net/ipv4/checkpoint.c +10
-0+ }else { +ipv6_addr_copy (&inet6->saddr, &hh->inet6.saddr); +ipv6_addr_copy (&inet6->rcv_saddr, &hh->inet6.rcv_saddr); +ipv6_addr_copy (&inet6->daddr, &hh->inet6.daddr); + } + } + +return ret; +} +
[...]
Oren.
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
OL> * Did you test this with UDP too ?
Not sendmail of course, but I have a little test program that maintains a DGRAM connection to the echo service on a remote node, yeah.
OL> * What happens if the the clock on the target machine differs from OL> the clock on the origin machine ? (TCP timestamps)
I guess maybe we should canonicalize the timeout values to something like "milliseconds after checkpoint start"? This would allow the remote system to reset the timers to something reasonable. It would also cause non-migration restarts to restore the timers appropriately for a coordinated restart of multiple machines.
OL> * How confident are we that "bad" input in one or more fields, OL> that you don't currently sanitize, cannot create "bad" behavior ? OL> (bad can be kernel crash, unauthorized behavior, DoS etc)
I'm going to say 0.052.
I haven't evaluated much of it, no :)
OL> * How much does TCP rely on the validity of the info in the OL> protocol control block, and what sorts of bads can happen if it OL> isn't ? Would TCP be still happy if the URG point is bogus, would OL> it allow the user to sent packets otherwise disallowed (to that OL> user?), or maybe it could crash the kernel ?
Good question, I'll have to look.
OL> * Can you please document (brief description) how the restart OL> logic works (listening parent socket etc) ?
Sure.
OL> * Do you intend to checkpoint (and collect) lingering sockets, OL> that is they are closed by the application so not references by OL> any task, but still sending data from their buffers ?
Yeah, I expect that will be important :)
OL> * I'd like to also preserve the "older" behavior - so the user can OL> choose to restart and reset all previous connections, keep OL> listening sockets (e.g. RESTART_DISCONNET).
Sure, sounds good to me.
Quoted 1 line
net/ipv4/checkpoint.c struct dq_buffers { · line 47
Quoted 1 line
net/ipv4/checkpoint.c +1
-0+printk ("Doing post-restart hash\n" );
(oops, looks like I left some debug messages in place)
OL> I wonder if a user can use this to convince TCP to send some nasty OL> packets to some arbitrary destination, with specific seq-number or OL> what not ?
I'm not sure what you mean. The sk->num value comes from the sport which should have been refused during the bind() if it's in use or not permitted, no?
Thanks!
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: [email protected]
A message that is not in the archive (sent off-list or not yet received): 87ws2zcjhe.fsf-FLMGYpZoEPULwtHQx/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 6 lines · Dan Smith
OL> * Did you test this with UDP too ?
Not sendmail of course, but I have a little test program that maintains a DGRAM connection to the echo service on a remote node, yeah.
Quoted 8 lines
OL> * What happens if the the clock on the target machine differs from OL> the clock on the origin machine ? (TCP timestamps)
I guess maybe we should canonicalize the timeout values to something like "milliseconds after checkpoint start"? This would allow the remote system to reset the timers to something reasonable. It would also cause non-migration restarts to restore the timers appropriately for a coordinated restart of multiple machines.
IIRC, the TCP stack takes the timestamp for each packet directly from jiffies. So you need to teach TCP to add a per-container (or you can make it per-socket) delta to that timestamp.
Quoted 6 lines
OL> * How confident are we that "bad" input in one or more fields, OL> that you don't currently sanitize, cannot create "bad" behavior ? OL> (bad can be kernel crash, unauthorized behavior, DoS etc)
I'm going to say 0.052.
Ah ... sure ...
To avoid confusion, can you state the units :p
Quoted 2 lines
I haven't evaluated much of it, no :)
I guess my point is that we want to ask the networking people this question in an explicit way.
Quoted 8 lines
OL> * How much does TCP rely on the validity of the info in the OL> protocol control block, and what sorts of bads can happen if it OL> isn't ? Would TCP be still happy if the URG point is bogus, would OL> it allow the user to sent packets otherwise disallowed (to that OL> user?), or maybe it could crash the kernel ?
Good question, I'll have to look.
Ditto.
So I'm thinking, for both, do (1) put a big fat comment in the code saying that sanity-tests are needed, and what for, and (2) send a separate mail to the networking people with these two scenarios and request comments ?
Quoted 11 lines
OL> * Can you please document (brief description) how the restart OL> logic works (listening parent socket etc) ?
Sure.
OL> * Do you intend to checkpoint (and collect) lingering sockets, OL> that is they are closed by the application so not references by OL> any task, but still sending data from their buffers ?
Yeah, I expect that will be important :)
Cool. How about a TODO comment somewhere to convince everyone ( = me) that you have it in your plans :)
Quoted 11 lines
OL> * I'd like to also preserve the "older" behavior - so the user can OL> choose to restart and reset all previous connections, keep OL> listening sockets (e.g.
RESTART_DISCONNET).Sure, sounds good to me.
Quoted 1 line
net/ipv4/checkpoint.c struct dq_buffers { · line 47
Quoted 1 line
net/ipv4/checkpoint.c +1
-0+printk ("Doing post-restart hash\n" );(oops, looks like I left some debug messages in place)
Quoted 7 lines
OL> I wonder if a user can use this to convince TCP to send some nasty OL> packets to some arbitrary destination, with specific seq-number or OL> what not ?
I'm not sure what you mean. The
sk->numvalue comes from the sport which should have been refused during thebind()if it's in use or not permitted, no?
I think Serge already pointed in his review that this should not permit a user to bind inconsistent or restricted ports.
I actually meant the contrary: suppose a malicious user on your machine wants to attack a target machine/connection. can that user provide such destination-address data and protocol parameters to build a connection that locally seems valid, but is malicious ?
For example, now, if a user wants to send a TCP packet with arbitrary protocol parameters, he needs to use raw IP sockets, which require privilege. Would restarting a connection with the desired parameters become a way to bypass that restriction ? (e.g. assume the user restarts while using the host's network namespace).
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
OL> IIRC, the TCP stack takes the timestamp for each packet directly OL> from jiffies. So you need to teach TCP to add a per-container (or OL> you can make it per-socket) delta to that timestamp.
After wondering what the heck you were talking about, I realized I assumed you were talking about TCP timeouts and not timestamps :)
I assume you mean the following:
1. Put a absolute time stamp in the checkpoint stream
2. Calculate the delta between that and the current time on the restoring host
3. Use that value to offset timestamps from that point on.
Correct?
Since I brought it up, do you agree that the retransmit timers should be canonicalized to time-after checkpoint values? It occurs to me that right now I restore a jiffies value on the receiving host which is guaranteed to be incorrect :)
OL> So I'm thinking, for both, do (1) put a big fat comment in the OL> code saying that sanity-tests are needed, and what for, and (2) OL> send a separate mail to the networking people with these two OL> scenarios and request comments ?
Yeah, although I would hope that they're seeing this conversation and would chime in (hence the cc:netdev). Hopefully I don't have to disguise a separate email as non-C/R related to get past their filters! :)
OL> For example, now, if a user wants to send a TCP packet with OL> arbitrary protocol parameters, he needs to use raw IP sockets, OL> which require privilege. Would restarting a connection with the OL> desired parameters become a way to bypass that restriction ? OL> (e.g. assume the user restarts while using the host's network OL> namespace).
Um, yeah? I don't see much way around that if we're going to trust any of what is in the checkpoint stream. Perhaps we say CAP_NET_ADMIN is required to restart a live TCP connection?
Signature · 3 lines
Dan Smith IBM Linux Technology Center email: danms-r/[email protected]
Re: [PATCH 2/2] [RFC] Add c/r support for connected INET sockets
Quoted 7 lines · Dan Smith
OL> IIRC, the TCP stack takes the timestamp for each packet directly OL> from jiffies. So you need to teach TCP to add a per-container (or OL> you can make it per-socket) delta to that timestamp.
After wondering what the heck you were talking about, I realized I assumed you were talking about TCP timeouts and not timestamps :)
Quoted 8 lines
I assume you mean the following:
1. Put a absolute time stamp in the checkpoint stream
2. Calculate the delta between that and the current time on the restoring host
3. Use that value to offset timestamps from that point on.Correct?
Sort of. Right now we already record the absolute time-of-checkpoint:ctx->ktime_begin. The restart-blocks are saved relative to it. I'd suggest the same for all time related data from the network - save it in the checkpoint image as delta's compared to checkpoint time.
At restart, the restart-blocks are restored relative to restart-time, using the saved delta. That would work for the TCP timestamps too.
Quoted 5 lines
Since I brought it up, do you agree that the retransmit timers should be canonicalized to time-after checkpoint values? It occurs to me that right now I restore a jiffies value on the receiving host which is guaranteed to be incorrect :)
As for TCP timeouts - I don't think they matter that much in the case of live migration, whether the timeout after restart happens in the saved delta relative to original checkpoint-time, or new restart-time. The difference is likely to be subsecond to a few seconds at most, not important for most use-cases, I'd think.
(If we are concerned about a TCP hickup due to a migration, there are tricks to work around it that; timeouts and retransmits are not the best way to go, because once you get there you already slowed down TCP significantly).
Here, too, once we have time virtualization this can be revisited, to allow the user to choose a policy how to use the deltas.
Quoted 18 lines
OL> So I'm thinking, for both, do (1) put a big fat comment in the OL> code saying that sanity-tests are needed, and what for, and (2) OL> send a separate mail to the networking people with these two OL> scenarios and request comments ?
Yeah, although I would hope that they're seeing this conversation and would chime in (hence the cc:netdev). Hopefully I don't have to disguise a separate email as non-C/R related to get past their filters! :)
OL> For example, now, if a user wants to send a TCP packet with OL> arbitrary protocol parameters, he needs to use raw IP sockets, OL> which require privilege. Would restarting a connection with the OL> desired parameters become a way to bypass that restriction ? OL> (e.g. assume the user restarts while using the host's network OL> namespace).
Quoted 4 lines
Um, yeah? I don't see much way around that if we're going to trust any of what is in the checkpoint stream. Perhaps we say
CAP_NET_ADMINis required to restart a live TCP connection?
I don't see much way around that either. My point is to bring the issue to everyone's attention, and see what others say about it.
CAP_NET_ADMIN is one option. CAP_NET_RAW is another option ?
Oren.