Add support for tcp communications [38/70]
From fdda387f73947e6ae511ec601f5b3c6fbb582aac Mon Sep 17 00:00:00 2001
From: Patrick Caulfield <[email protected]>
Date: Thu, 2 Nov 2006 11:19:21 -0500
Subject: [PATCH] [DLM] Add support for tcp communications
The following patch adds a TCP based communications layer to the DLM which is compile time selectable. The existing SCTP layer gives the advantage of allowing multihoming, whereas the TCP layer has been heavily tested in previous versions of the DLM and is known to be robust and therefore can be used as a baseline for performance testing.
--- fs/dlm/Kconfig | 17 + fs/dlm/Makefile | 4 fs/dlm/lowcomms-sctp.c | 1239 +++++++++++++++++++++++++++++++++++++++++++++++ fs/dlm/lowcomms-tcp.c | 1263 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/dlm/lowcomms.c | 1239 ----------------------------------------------- 5 files changed, 2522 insertions(+), 1240 deletions(-)
fs/dlm/Kconfig +17 -0
@@ -9,6 +9,23 @@ config DLM A general purpose distributed lock manager for kernel or userspace applications. +choice + prompt "Select DLM communications protocol" + depends on DLM + default DLM_TCP + help + The DLM Can use TCP or SCTP for it's network communications. + SCTP supports multi-homed operations whereas TCP doesn't. + However, SCTP seems to have stability problems at the moment. + +config DLM_TCP + bool "TCP/IP" + +config DLM_SCTP + bool "SCTP" + +endchoice + config DLM_DEBUG bool "DLM debugging" depends on DLM
fs/dlm/Makefile +3 -1
@@ -4,7 +4,6 @@ dlm-y := ast.o \ dir.o \ lock.o \ lockspace.o \ - lowcomms.o \ main.o \ member.o \ memory.o \
@@ -17,3 +16,6 @@ dlm-y := ast.o \ util.o dlm-$(CONFIG_DLM_DEBUG) += debug_fs.o +dlm-$(CONFIG_DLM_TCP) += lowcomms-tcp.o + +dlm-$(CONFIG_DLM_SCTP) += lowcomms-sctp.o
fs/dlm/lowcomms-sctp.c new file +1239 -0
1,239 lines not on this page · raw message
fs/dlm/lowcomms-tcp.c new file +1263 -0
1,263 lines not on this page · raw message
fs/dlm/lowcomms.c deleted +0 -1239
461 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.4.1
Part of the diff is not shown here. View the raw message.
Hi!
Quoted 11 lines
From: Patrick Caulfield <[email protected]>
Date: Thu, 2 Nov 2006 11:19:21 -0500
Subject: [PATCH] [DLM] Add support for tcp communicationsThe following patch adds a TCP based communications layer to the DLM which is compile time selectable. The existing SCTP layer gives the advantage of allowing multihoming, whereas the TCP layer has been heavily tested in previous versions of the DLM and is known to be robust and therefore can be used as a baseline for performance testing.
Quoted 2 lines
Signed-off-by Patrick Caulfield <[email protected]>Signed-off-by Steven Whitehouse <[email protected]>
Some comments from coding-style-police...
fs/dlm/lowcomms-sctp.c line 85
Quoted 5 lines
fs/dlm/lowcomms-sctp.c new file +5
-0+struct cbuf { +unsigned base; +unsigned len; +unsigned mask; +};
I'd prefer unsigned int here.
fs/dlm/lowcomms-sctp.c line 99
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+struct connection { +struct socket *sock; +unsigned long flags; +struct page *rx_page; +atomic_t waiting_requests; +struct cbuf cb; +int eagain_flag; +};
Quite unusual indentiation style w.r.t. pointers...
Quoted 16 lines
fs/dlm/lowcomms-sctp.c new file +16
-0+static struct nodeinfo *nodeid2nodeinfo (int nodeid,gfp_t alloc) +{ +struct nodeinfo *ni; +int r; +int n; + +down_read (&nodeinfo_lock); + ni =idr_find (&nodeinfo_idr, nodeid); +up_read (&nodeinfo_lock); + +if (!ni && alloc) { +down_write (&nodeinfo_lock); + + ni =idr_find (&nodeinfo_idr, nodeid); +if (ni) + goto out_up;
...
fs/dlm/lowcomms-sctp.c line 226
Quoted 7 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+if (nodeid > max_nodeid) + max_nodeid = nodeid; + out_up: +up_write (&nodeinfo_lock); + } + +return ni;
Can we do if (ni || !alloc) return ni, and avoid gotos inside of block?
Quoted 11 lines
fs/dlm/lowcomms-sctp.c new file +11
-0+/* Something happened to an association */ +static void process_sctp_notification (struct msghdr *msg,char *buf) +{ +union sctp_notification *sn = (union sctp_notification *)buf; + +if (sn->sn_header.sn_type ==SCTP_ASSOC_CHANGE ) { +switch (sn->sn_assoc_change.sac_state) { + +case SCTP_COMM_UP : +case SCTP_RESTART : + {
...
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Retry INIT later */ + ni = > assoc2nodeinfo(sn->sn_assoc_change.sac_assoc_id);
Too long lines, and rgetting rid of outer if should be really easy.
fs/dlm/lowcomms-sctp.c line 839
Quoted 5 lines
fs/dlm/lowcomms-sctp.c new file +5
-0+ +/* Initiate an SCTP association. In theory we could just use sendmsg() on +the first IP address and it should work, but this allows us to set up the +association before sending any valuable data that we can't afford to lose. +It also keeps the send path clean as it can now always use the association ID */
Too long lines.
fs/dlm/lowcomms-tcp.c line 235
Quoted 3 lines
fs/dlm/lowcomms-tcp.c new file +3
-0+static void lowcomms_state_change (struct sock *sk) +{ +/* struct connection *con = sock2con(sk); */
Can we get commented code removed?
Quoted 8 lines
fs/dlm/lowcomms-tcp.c new file +8
-0+ +switch (sk->sk_state) { +case TCP_ESTABLISHED : +lowcomms_write_space (sk); +break ; + +case TCP_FIN_WAIT1 : +case TCP_FIN_WAIT2 :
fs/dlm/lowcomms-tcp.c line 253
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+case TCP_TIME_WAIT : +case TCP_CLOSE : +case TCP_CLOSE_WAIT : +case TCP_LAST_ACK : +case TCP_CLOSING : +/* FIXME: I think this causes more trouble than it solves. +lowcomms wil reconnect anyway when there is something to +send. This just attempts reconnection if a node goes down! +*/ +/* lowcomms_connect_sock(con); */
Here too? Is the fixme still applicable?
fs/dlm/lowcomms-tcp.c line 286
Quoted 8 lines
fs/dlm/lowcomms-tcp.c new file +8
-0+{ + saddr->ss_family = dlm_local_addr.ss_family; +if (saddr->ss_family ==AF_INET ) { +struct sockaddr_in *in4_addr = (struct sockaddr_in *)saddr; + in4_addr->sin_port =cpu_to_be16 (port); + *addr_len =sizeof (struct sockaddr_in ); + } +else {
} else { on one line, please.
...
Quoted 5 lines
fs/dlm/lowcomms-tcp.c new file +5
-0+if (ret <0 ) + goto out_close; +CBUF_EAT (&con->cb, ret); + +if (CBUF_EMPTY (&con->cb) && !call_again_soon) {
fs/dlm/lowcomms-tcp.c line 406
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+__free_page (con->rx_page); + con->rx_page =NULL ; + } + + out: +if (call_again_soon) + goto out_resched; +up_read (&con->sock_sem); + ret =0 ; + goto out_ret;
Could we just return 0 here?
fs/dlm/lowcomms-tcp.c line 413
Quoted 6 lines
fs/dlm/lowcomms-tcp.c new file +6
-0+ out_resched: +lowcomms_data_ready (con->sock->sk,0 ); +up_read (&con->sock_sem); + ret =0 ; +schedule (); + goto out_ret;
And here?
Quoted 7 lines
fs/dlm/lowcomms-tcp.c new file +7
-0+ out_close: +up_read (&con->sock_sem); +if (ret != -EAGAIN && !test_bit (CF_IS_OTHERCON , &con->flags)) { +close_connection (con, FALSE); +/* Reconnect when there is something to send */ + } +
fs/dlm/lowcomms-tcp.c
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +9
-0+ out_ret: +return ret; +static struct socket *create_listen_sock (struct connection *con,struct sockaddr_storage *saddr) +{ +struct socket *sock =NULL ; +mm_segment_t fs; +int result =0 ; +int one =1 ; +int addr_len;
Tabs vs spaces?
fs/dlm/lowcomms-tcp.c line 671
Quoted 11 lines
fs/dlm/lowcomms-tcp.c new file +11
-0+ result = sock->ops->listen (sock,5 ); +if (result <0 ) { +printk ("dlm: Can't listen on port %d\n" , dlm_config.tcp_port); +sock_release (sock); + sock =NULL ; + goto create_out; + } + + create_out: +return sock; +}
I seriously think you are overdoing the gotos.
Quoted 11 lines
fs/dlm/lowcomms-tcp.c new file +11
-0+/* Called from recovery when it knows that a node has +left the cluster */ +int dlm_lowcomms_close (int nodeid) +{ +struct connection *con; + +if (!connections) + goto out; + +log_print ("closing connection to node %d" , nodeid); + con =nodeid2con (nodeid,0 );
fs/dlm/lowcomms-tcp.c line 914
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+if (con) { +clean_one_writequeue (con); +close_connection (con, TRUE); +atomic_set (&con->waiting_requests,0 ); + } +return 0 ; + + out: +return -1 ; +}
Could we just return -1 directly? Should it follow 0/-errno convention other functions here use? Can we get rid of TRUE/FALSE macros?
fs/dlm/lowcomms-tcp.c line 1138
Quoted 7 lines
fs/dlm/lowcomms-tcp.c new file +7
-0+/* +* Return the largest buffer size we can cope with. +*/ +int lowcomms_max_buffer_size (void ) +{ +return PAGE_CACHE_SIZE ; +}
Uhuh.
fs/dlm/lowcomms-tcp.c line 1176
Quoted 6 lines
fs/dlm/lowcomms-tcp.c new file +6
-0+/* This is quite likely to sleep... */ +int dlm_lowcomms_start (void ) +{ +int error =0 ; + + error = -ENOTCONN ;
One statement?
fs/dlm/lowcomms-tcp.c line 1192
Quoted 7 lines
fs/dlm/lowcomms-tcp.c new file +7
-0+ connections =kmalloc (sizeof (struct connection *) * +NODE_INCREMENT ,GFP_KERNEL ); +if (!connections) + goto out; + +memset (connections,0 , +sizeof (struct connection *) *NODE_INCREMENT );
kzalloc?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 5 lines · On Thu, 30 Nov 2006 12:19:08 +0000 Steven Whitehouse <[email protected]>…
Quoted 1 line
From
fdda387f73947e6ae511ec601f5b3c6fbb582aacMon Sep 17 00:00:00 2001From: Patrick Caulfield <[email protected]>
Date: Thu, 2 Nov 2006 11:19:21 -0500
Subject: [PATCH] [DLM] Add support for tcp communications
Quoted 7 lines
The following patch adds a TCP based communications layer to the DLM which is compile time selectable. The existing SCTP layer gives the advantage of allowing multihoming, whereas the TCP layer has been heavily tested in previous versions of the DLM and is known to be robust and therefore can be used as a baseline for performance testing.
I don't think this code is ready.
Quoted 3 lines
...
+static struct
rw_semaphorenodeinfo_lock;
This can be initialised at compile-time.
Quoted 9 lines
...
fs/dlm/lowcomms-sctp.c new file +7
-0+#define CBUF_ADD (cb, n) do { (cb)->len += n; } while(0) +#define CBUF_EMPTY (cb) ((cb)->len == 0) +#define CBUF_MAY_ADD (cb, n) (((cb)->len + (n)) < ((cb)->mask + 1)) +#define CBUF_DATA (cb) (((cb)->base + (cb)->len) & (cb)->mask) + +#define CBUF_INIT (cb, size) \ +do { \
fs/dlm/lowcomms-sctp.c line 129
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ (cb)->base = (cb)->len =0 ; \ + (cb)->mask = ((size)-1 ); \ +} while(0) + +#define CBUF_EAT (cb, n) \ +do { \ + (cb)->len -= (n); \ + (cb)->base += (n); \ + (cb)->base &= (cb)->mask; \ +} while(0)
Suggest that all of the above be converted to C: lower-case static-inline functions.
fs/dlm/lowcomms-sctp.c line 133
Quoted 3 lines
fs/dlm/lowcomms-sctp.c new file +3
-0+ +/* List of nodes which have writes pending */ +static struct list_head write_nodes;
LIST_HEAD (maybe)
Quoted 1 line
+static
spinlock_twrite_nodes_lock;
DEFINE_SPINLOCK
fs/dlm/lowcomms-sctp.c line 144
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ +/* Maximum number of incoming messages to process before +* doing a schedule() +*/ +#define MAX_RX_MSG_COUNT 25 + +/* Manage daemons */ +static struct task_struct *recv_task; +static struct task_struct *send_task; +static wait_queue_head_t lowcomms_recv_wait;
DECLARE_WAIT_QUEUE_HEAD
Quoted 1 line
+static
atomic_taccepting;
= ATOMIC_INIT(0)
Generally, initialising things at compile-time is better: smaller vmlinux, certainty that the code got it right.
fs/dlm/lowcomms-sctp.c line 183
Quoted 7 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+static struct nodeinfo *nodeid2nodeinfo (int nodeid,gfp_t alloc) +{ +struct nodeinfo *ni; +int r; +int n; + +down_read (&nodeinfo_lock);
Given that this function can sleep, I wonder if `alloc' is useful.
I see lots of callers passing in a literal "0" for `alloc'. That's in fact a secret (GFP_ATOMIC & ~__GFP_HIGH). I doubt if that's what you really meant. Particularly as the code could at least have used __GFP_WAIT (aka GFP_NOIO) which is much, much more reliable than "0". In fact "0" is the least reliable mode possible.
IOW, this is all bollixed up.
Quoted 5 lines
fs/dlm/lowcomms-sctp.c new file +5
-0+ ni =idr_find (&nodeinfo_idr, nodeid); +up_read (&nodeinfo_lock); + +if (!ni && alloc) { +down_write (&nodeinfo_lock);
fs/dlm/lowcomms-sctp.c line 198
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ + ni =idr_find (&nodeinfo_idr, nodeid); +if (ni) + goto out_up; + + r =idr_pre_get (&nodeinfo_idr, alloc); +if (!r) + goto out_up; + + ni =kmalloc (sizeof (struct nodeinfo ), alloc);
kzalloc
Quoted 6 lines
fs/dlm/lowcomms-sctp.c new file +6
-0+if (!ni) + goto out_up; + + r =idr_get_new_above (&nodeinfo_idr, ni, nodeid, &n); +if (r) { +kfree (ni);
fs/dlm/lowcomms-sctp.c line 214
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ ni =NULL ; + goto out_up; + } +if (n != nodeid) { +idr_remove (&nodeinfo_idr, n); +kfree (ni); + ni =NULL ; + goto out_up; + } +memset (ni,0 ,sizeof (struct nodeinfo ));
hence nuke
Quoted 12 lines
fs/dlm/lowcomms-sctp.c new file +12
-0+spin_lock_init (&ni->lock); +INIT_LIST_HEAD (&ni->writequeue); +spin_lock_init (&ni->writequeue_lock); + ni->nodeid = nodeid; + +if (nodeid > max_nodeid) + max_nodeid = nodeid; + out_up: +up_write (&nodeinfo_lock); + } + +return ni;
fs/dlm/lowcomms-sctp.c line 236
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+} + +/* Don't call this too often... */ +static struct nodeinfo *assoc2nodeinfo (sctp_assoc_t assoc) +{ +int i; +struct nodeinfo *ni; + +for (i=1 ; i<=max_nodeid; i++) { + ni =nodeid2nodeinfo (i,0 );
GFP_NOIO (at least) (lots of places)
Quoted 19 lines
fs/dlm/lowcomms-sctp.c new file +16
-0+if (ni && ni->assoc_id == assoc) +return ni; + } +return NULL ; +} + +static void send_shutdown (sctp_assoc_t associd) +{ +static char outcmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))]; +struct msghdr outmessage; +struct cmsghdr *cmsg; +struct sctp_sndrcvinfo *sinfo; +int ret; + + outmessage.msg_name =NULL ; + outmessage.msg_namelen =0 ;
fs/dlm/lowcomms-sctp.c line 327
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ outmessage.msg_control = outcmsg; + outmessage.msg_controllen =sizeof (outcmsg); + outmessage.msg_flags =MSG_EOR ; + + cmsg =CMSG_FIRSTHDR (&outmessage); + cmsg->cmsg_level =IPPROTO_SCTP ; + cmsg->cmsg_type =SCTP_SNDRCV ; + cmsg->cmsg_len =CMSG_LEN (sizeof (struct sctp_sndrcvinfo )); + outmessage.msg_controllen = cmsg->cmsg_len; + sinfo = (struct sctp_sndrcvinfo *)CMSG_DATA (cmsg);
CMSG_DATA() returns void* - unneeded cast
Quoted 11 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+memset (sinfo,0x00 ,sizeof (struct sctp_sndrcvinfo )); + + sinfo->sinfo_flags |=MSG_EOF ; + sinfo->sinfo_assoc_id = associd; + + ret =kernel_sendmsg (sctp_con.sock, &outmessage,NULL ,0 ,0 ); + +if (ret !=0 ) +log_print ("send EOF to node failed: %d" , ret); +}
fs/dlm/lowcomms-sctp.c line 431
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+ ni =nodeid2nodeinfo (nodeid,GFP_KERNEL ); +if (!ni) +return ; + +/* Save the assoc ID */ +spin_lock (&ni->lock); + ni->assoc_id = sn->sn_assoc_change.sac_assoc_id; +spin_unlock (&ni->lock);
A bare assignment inside a lock like this is quite unusual. It is often an indication that something is wrong.
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+/* Data received from remote end */ +static int receive_from_sock (void ) +{ +int ret =0 ; +struct msghdr msg; +struct kvec iov[2 ]; +unsigned len; +int r;
fs/dlm/lowcomms-sctp.c line 497
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+struct sctp_sndrcvinfo *sinfo; +struct cmsghdr *cmsg; +struct nodeinfo *ni; + +/* These two are marginally too big for stack allocation, but this +* function is (currently) only called by dlm_recvd so static should be +* OK. +*/ +static struct sockaddr_storage msgname; +static char incmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];
whoa. This is globally singly-threaded code??
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+if (sctp_con.sock ==NULL ) + goto out; + +if (sctp_con.rx_page ==NULL ) { +/* +* This doesn't need to be atomic, but I think it should +* improve performance if it is. +*/
fs/dlm/lowcomms-sctp.c line 516
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ sctp_con.rx_page =alloc_page (GFP_ATOMIC ); +if (sctp_con.rx_page ==NULL ) + goto out_resched; +CBUF_INIT (&sctp_con.cb,PAGE_CACHE_SIZE ); + } + +memset (&incmsg,0 ,sizeof (incmsg)); +memset (&msgname,0 ,sizeof (msgname)); + +memset (incmsg,0 ,sizeof (incmsg));
You just zeroed it twice.
Quoted 69 lines
fs/dlm/lowcomms-sctp.c new file +69
-0+ msg.msg_name = &msgname; + msg.msg_namelen =sizeof (msgname); + msg.msg_flags =0 ; + msg.msg_control = incmsg; + msg.msg_controllen =sizeof (incmsg); + msg.msg_iovlen =1 ; + +/* I don't see why this circular buffer stuff is necessary for SCTP +* which is a packet-based protocol, but the whole thing breaks under +* load without it! The overhead is minimal (and is in the TCP lowcomms +* anyway, of course) so I'll leave it in until I can figure out what's +* really happening. +*/ + +/* +* iov[0] is the bit of the circular buffer between the current end +* point (cb.base + cb.len) and the end of the buffer. +*/ + iov[0 ].iov_len = sctp_con.cb.base -CBUF_DATA (&sctp_con.cb); + iov[0 ].iov_base =page_address (sctp_con.rx_page) + +CBUF_DATA (&sctp_con.cb); + iov[1 ].iov_len =0 ; + +/* +* iov[1] is the bit of the circular buffer between the start of the +* buffer and the start of the currently used section (cb.base) +*/ +if (CBUF_DATA (&sctp_con.cb) >= sctp_con.cb.base) { + iov[0 ].iov_len =PAGE_CACHE_SIZE -CBUF_DATA (&sctp_con.cb); + iov[1 ].iov_len = sctp_con.cb.base; + iov[1 ].iov_base =page_address (sctp_con.rx_page); + msg.msg_iovlen =2 ; + } + len = iov[0 ].iov_len + iov[1 ].iov_len; + + r = ret =kernel_recvmsg (sctp_con.sock, &msg, iov, msg.msg_iovlen, len, +MSG_NOSIGNAL |MSG_DONTWAIT ); +if (ret <=0 ) + goto out_close; + + msg.msg_control = incmsg; + msg.msg_controllen =sizeof (incmsg); + cmsg =CMSG_FIRSTHDR (&msg); + sinfo = (struct sctp_sndrcvinfo *)CMSG_DATA (cmsg); + +if (msg.msg_flags &MSG_NOTIFICATION ) { +process_sctp_notification (&msg,page_address (sctp_con.rx_page)); +return 0 ; + } + +/* Is this a new association ? */ + ni =nodeid2nodeinfo (le32_to_cpu (sinfo->sinfo_ppid),GFP_KERNEL ); +if (ni) { + ni->assoc_id = sinfo->sinfo_assoc_id; +if (test_and_clear_bit (NI_INIT_PENDING , &ni->flags)) { + +if (!test_and_set_bit (NI_WRITE_PENDING , &ni->flags)) { +spin_lock_bh (&write_nodes_lock); +list_add_tail (&ni->write_list, &write_nodes); +spin_unlock_bh (&write_nodes_lock); + } +wake_up_process (send_task); + } + } + +/* INIT sends a message with length of 1 - ignore it */ +if (r ==1 ) +return 0 ; +
fs/dlm/lowcomms-sctp.c line 595
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+CBUF_ADD (&sctp_con.cb, ret); + ret =dlm_process_incoming_buffer (cpu_to_le32 (sinfo->sinfo_ppid), +page_address (sctp_con.rx_page), + sctp_con.cb.base, sctp_con.cb.len, +PAGE_CACHE_SIZE ); +if (ret <0 ) + goto out_close; +CBUF_EAT (&sctp_con.cb, ret); + + out:
Labels go in column 0 or 1
fs/dlm/lowcomms-sctp.c line 602
Quoted 7 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+ ret =0 ; + goto out_ret; + + out_resched: +lowcomms_data_ready (sctp_con.sock->sk,0 ); + ret =0 ; +schedule ();
What's the random schedule() for? If !need_reached() it's just a waste of cycles.
Quoted 11 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+ goto out_ret; + + out_close: +if (ret != -EAGAIN ) +log_print ("error reading from sctp socket: %d" , ret); + out_ret: +return ret; +} +
fs/dlm/lowcomms-sctp.c line 760
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+void *dlm_lowcomms_get_buffer (int nodeid,int len,gfp_t allocation,char **ppc) +{ +struct writequeue_entry *e; +int offset =0 ; +int users =0 ; +struct nodeinfo *ni; + +if (!atomic_read (&accepting)) +return NULL ;
hm, this looks racy. What happens if `accepting' goes to zero 50 nanoseconds later?
fs/dlm/lowcomms-sctp.c line 768
Quoted 7 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+ ni =nodeid2nodeinfo (nodeid, allocation); +if (!ni) +return NULL ; + +spin_lock (&ni->writequeue_lock); + e = list_entry(ni->writequeue.prev, struct writequeue_entry, list); + if (((struct list_head *) e == &ni->writequeue) ||
That typecast looks fishy. Perhaps a container_of() or something would be clearer.
fs/dlm/lowcomms-sctp.c line 779
Quoted 11 lines
fs/dlm/lowcomms-sctp.c new file +11
-0+ (PAGE_CACHE_SIZE - e->end < len)) { + e = NULL; + } else { + offset = e->end; + e->end += len; + users = e->users++; + } +spin_unlock (&ni->writequeue_lock); + +if (e) { + got_one:
whoa, how'd that label get all the way over there?
Quoted 17 lines
fs/dlm/lowcomms-sctp.c new file +17
-0+if (users ==0 ) +kmap (e->page); + *ppc =page_address (e->page) + offset; +return e; + } + + e =new_writequeue_entry (allocation); +if (e) { +spin_lock (&ni->writequeue_lock); + offset = e->end; + e->end += len; + e->ni = ni; + users = e->users++; +list_add_tail (&e->list, &ni->writequeue); +spin_unlock (&ni->writequeue_lock); + goto got_one; + }
fs/dlm/lowcomms-sctp.c line 843
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+return NULL ; +} + +static void initiate_association (int nodeid) +{ +struct sockaddr_storage rem_addr; +static char outcmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];
Another static buffer to worry about. Globally singly-threaded code?
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+struct msghdr outmessage; +struct cmsghdr *cmsg; +struct sctp_sndrcvinfo *sinfo; +int ret; +int addrlen; +char buf[1 ]; +struct kvec iov[1 ]; +struct nodeinfo *ni;
fs/dlm/lowcomms-sctp.c line 861
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ +log_print ("Initiating association with node %d" , nodeid); + + ni =nodeid2nodeinfo (nodeid,GFP_KERNEL ); +if (!ni) +return ; + +if (nodeid_to_addr (nodeid, (struct sockaddr *)&rem_addr)) { +log_print ("no address for nodeid %d" , nodeid); +return ;
Am surprised that all these errors appear to be ignored.
Quoted 30 lines
fs/dlm/lowcomms-sctp.c new file +30
-0+ } + +make_sockaddr (&rem_addr, dlm_config.tcp_port, &addrlen); + + outmessage.msg_name = &rem_addr; + outmessage.msg_namelen = addrlen; + outmessage.msg_control = outcmsg; + outmessage.msg_controllen =sizeof (outcmsg); + outmessage.msg_flags =MSG_EOR ; + + iov[0 ].iov_base = buf; + iov[0 ].iov_len =1 ; + +/* Real INIT messages seem to cause trouble. Just send a 1 byte message +we can afford to lose */ + cmsg =CMSG_FIRSTHDR (&outmessage); + cmsg->cmsg_level =IPPROTO_SCTP ; + cmsg->cmsg_type =SCTP_SNDRCV ; + cmsg->cmsg_len =CMSG_LEN (sizeof (struct sctp_sndrcvinfo )); + sinfo = (struct sctp_sndrcvinfo *)CMSG_DATA (cmsg); +memset (sinfo,0x00 ,sizeof (struct sctp_sndrcvinfo )); + sinfo->sinfo_ppid =cpu_to_le32 (dlm_local_nodeid); + + outmessage.msg_controllen = cmsg->cmsg_len; + ret =kernel_sendmsg (sctp_con.sock, &outmessage, iov,1 ,1 ); +if (ret <0 ) { +log_print ("send INIT to node failed: %d" , ret); +/* Try again later */ +clear_bit (NI_INIT_PENDING , &ni->flags); + }
fs/dlm/lowcomms-sctp.c line 901
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+} + +/* Send a message */ +static int send_to_sock (struct nodeinfo *ni) +{ +int ret =0 ; +struct writequeue_entry *e; +int len, offset; +struct msghdr outmsg; +static char outcmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];
Singly-threaded?
fs/dlm/lowcomms-sctp.c line 907
Quoted 6 lines
fs/dlm/lowcomms-sctp.c new file +6
-0+struct cmsghdr *cmsg; +struct sctp_sndrcvinfo *sinfo; +struct kvec iov; + +/* See if we need to init an association before we start +sending precious messages */
Please use tabs.
Quoted 9 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+spin_lock (&ni->lock); +if (!ni->assoc_id && !test_and_set_bit (NI_INIT_PENDING , &ni->flags)) { +spin_unlock (&ni->lock); +initiate_association (ni->nodeid); +return 0 ; + } +spin_unlock (&ni->lock); + + outmsg.msg_name =NULL ;/* We use assoc_id */
fs/dlm/lowcomms-sctp.c line 926
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ outmsg.msg_namelen =0 ; + outmsg.msg_control = outcmsg; + outmsg.msg_controllen =sizeof (outcmsg); + outmsg.msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL |MSG_EOR ; + + cmsg =CMSG_FIRSTHDR (&outmsg); + cmsg->cmsg_level =IPPROTO_SCTP ; + cmsg->cmsg_type =SCTP_SNDRCV ; + cmsg->cmsg_len =CMSG_LEN (sizeof (struct sctp_sndrcvinfo )); + sinfo = (struct sctp_sndrcvinfo *)CMSG_DATA (cmsg);
Unneeded cast
Quoted 6 lines
fs/dlm/lowcomms-sctp.c new file +6
-0+memset (sinfo,0x00 ,sizeof (struct sctp_sndrcvinfo )); + sinfo->sinfo_ppid =cpu_to_le32 (dlm_local_nodeid); + sinfo->sinfo_assoc_id = ni->assoc_id; + outmsg.msg_controllen = cmsg->cmsg_len; + +spin_lock (&ni->writequeue_lock);
fs/dlm/lowcomms-sctp.c line 942
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+for (;;) { +if (list_empty (&ni->writequeue)) +break ; + e = list_entry(ni->writequeue.next, struct writequeue_entry, + list); + len = e->len; + offset = e->offset; +BUG_ON (len ==0 && e->users ==0 ); +spin_unlock (&ni->writequeue_lock); +kmap (e->page);
I think this kmap() gets leaked
Quoted 44 lines
fs/dlm/lowcomms-sctp.c new file +43
-0+ ret =0 ; +if (len) { + iov.iov_base =page_address (e->page)+offset; + iov.iov_len = len; + + ret =kernel_sendmsg (sctp_con.sock, &outmsg, &iov,1 , + len); +if (ret == -EAGAIN ) { + sctp_con.eagain_flag =1 ; + goto out; + }else if (ret <0 ) + goto send_error; + }else { +/* Don't starve people filling buffers */ +schedule (); + } + +spin_lock (&ni->writequeue_lock); + e->offset += ret; + e->len -= ret; + +if (e->len ==0 && e->users ==0 ) { +list_del (&e->list); +free_entry (e); +continue ; + } + } +spin_unlock (&ni->writequeue_lock); + out: +return ret; + + send_error: +log_print ("Error sending to node %d %d" , ni->nodeid, ret); +spin_lock (&ni->lock); +if (!test_and_set_bit (NI_INIT_PENDING , &ni->flags)) { + ni->assoc_id =0 ; +spin_unlock (&ni->lock); +initiate_association (ni->nodeid); + }else +spin_unlock (&ni->lock); + +return ret; +}
fs/dlm/lowcomms-sctp.c line 1061
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+static void dealloc_nodeinfo (void ) +{ +int i; + +for (i=1 ; i<=max_nodeid; i++) { +struct nodeinfo *ni =nodeid2nodeinfo (i,0 ); +if (ni) { +idr_remove (&nodeinfo_idr, i);
Didn't that need locking?
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +5
-0+kfree (ni); + } + } +} +
fs/dlm/lowcomms-sctp.c line 1097
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+static int write_list_empty (void ) +{ +int status; + +spin_lock_bh (&write_nodes_lock); + status =list_empty (&write_nodes); +spin_unlock_bh (&write_nodes_lock); + +return status; +}
This function's return value is meaningless. As soon as the lock gets dropped, the return value can get out of sync with reality.
Looking at the caller, this _might_ happen to be OK, but it's a nasty and dangerous thing. Really the locking should be moved into the caller.
Quoted 13 lines
fs/dlm/lowcomms-sctp.c new file +13
-0+static int dlm_recvd (void *data) +{ +DECLARE_WAITQUEUE (wait, current); + +while (!kthread_should_stop ()) { +int count =0 ; + +set_current_state (TASK_INTERRUPTIBLE ); +add_wait_queue (&lowcomms_recv_wait, &wait); +if (!test_bit (CF_READ_PENDING , &sctp_con.flags)) +schedule (); +remove_wait_queue (&lowcomms_recv_wait, &wait); +set_current_state (TASK_RUNNING );
fs/dlm/lowcomms-sctp.c line 1121
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ +if (test_and_clear_bit (CF_READ_PENDING , &sctp_con.flags)) { +int ret; + + do { + ret = receive_from_sock(); + + /* Don't starve out everyone else */ + if (++count >= MAX_RX_MSG_COUNT) { + schedule();
cond_resched() would be more efficient. Potentially a lot more efficient. (several places).
Quoted 13 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+ count =0 ; + } + }while (!kthread_should_stop () && ret >=0 ); + } +schedule (); + } + +return 0 ; +} +static int daemons_start (void )
fs/dlm/lowcomms-sctp.c line 1172
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+{ +struct task_struct *p; +int error; + + p =kthread_run (dlm_recvd,NULL ,"dlm_recvd" ); + error =IS_ERR (p); +if (error) { +log_print ("can't start dlm_recvd %d" , error); +return error; + }
whitespace broke
fs/dlm/lowcomms-sctp.c line 1181
Quoted 9 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+ recv_task = p; + + p =kthread_run (dlm_sendd,NULL ,"dlm_sendd" ); + error =IS_ERR (p); +if (error) { +log_print ("can't start dlm_sendd %d" , error); +kthread_stop (recv_task); +return error; + }
ditto
fs/dlm/lowcomms-tcp.c line 65
Quoted 12 lines
fs/dlm/lowcomms-tcp.c new file +9
-0+ send_task = p; + +return 0 ; +} + +#ifndef FALSE +#define FALSE 0 +#define TRUE 1 +#endif
No, we have a kernel-wide bool/true/false implementation. Please use that.
fs/dlm/lowcomms-tcp.c line 74
Quoted 8 lines
fs/dlm/lowcomms-tcp.c new file +7
-0+#define CBUF_INIT (cb, size) do { (cb)->base = (cb)->len = 0; (cb)->mask = ((size)-1); } while(0) +#define CBUF_ADD (cb, n) do { (cb)->len += n; } while(0) +#define CBUF_EMPTY (cb) ((cb)->len == 0) +#define CBUF_MAY_ADD (cb, n) (((cb)->len + (n)) < ((cb)->mask + 1)) +#define CBUF_EAT (cb, n) do { (cb)->len -= (n); \ + (cb)->base += (n); (cb)->base &= (cb)->mask; } while(0) +#define CBUF_DATA (cb) (((cb)->base + (cb)->len) & (cb)->mask)
This cbuf API seems to be part-implemented in two places.
Ditto previous comments about macro->inline conversion
Quoted 20 lines
...
fs/dlm/lowcomms-tcp.c new file +19
-0+static struct sockaddr_storage dlm_local_addr; + +/* Manage daemons */ +static struct task_struct *recv_task; +static struct task_struct *send_task; + +static wait_queue_t lowcomms_send_waitq_head; +static wait_queue_head_t lowcomms_send_waitq; +static wait_queue_t lowcomms_recv_waitq_head; +static wait_queue_head_t lowcomms_recv_waitq; + +/* An array of pointers to connections, indexed by NODEID */ +static struct connection **connections; +static struct semaphore connections_lock; +static kmem_cache_t *con_cache; +static int conn_array_size; +static atomic_t accepting; + +/* List of sockets that have reads pending */
fs/dlm/lowcomms-tcp.c line 145
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+static struct list_head read_sockets; +static spinlock_t read_sockets_lock; + +/* List of sockets which have writes pending */ +static struct list_head write_sockets; +static spinlock_t write_sockets_lock; + +/* List of sockets which have connects pending */ +static struct list_head state_sockets; +static spinlock_t state_sockets_lock;
See previous comments regarding static initialisation
Quoted 5 lines
fs/dlm/lowcomms-tcp.c new file +5
-0+static struct connection *nodeid2con (int nodeid,gfp_t allocation) +{ +struct connection *con =NULL ; + +down (&connections_lock);
fs/dlm/lowcomms-tcp.c line 161
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+if (nodeid >= conn_array_size) { +int new_size = nodeid +NODE_INCREMENT ; +struct connection **new_conns; + + new_conns =kmalloc (sizeof (struct connection *) * + new_size, allocation); +if (!new_conns) + goto finish; + +memset (new_conns,0 ,sizeof (struct connection *) * new_size);
kzalloc()
Quoted 4 lines
fs/dlm/lowcomms-tcp.c new file +4
-0+memcpy (new_conns, connections,sizeof (struct connection *) * conn_array_size); + conn_array_size = new_size; +kfree (connections); + connections = new_conns;
fs/dlm/lowcomms-tcp.c line 175
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+ + } + + con = connections[nodeid]; +if (con ==NULL && allocation) { + con =kmem_cache_alloc (con_cache, allocation); +if (!con) + goto finish; + +memset (con,0 ,sizeof (*con));
kmem_cache_zalloc()
Quoted 16 lines
fs/dlm/lowcomms-tcp.c new file +13
-0+ con->nodeid = nodeid; +init_rwsem (&con->sock_sem); +INIT_LIST_HEAD (&con->writequeue); +spin_lock_init (&con->writequeue_lock); + + connections[nodeid] = con; + } + + finish: +up (&connections_lock); +return con; +} +/* Add the port number to an IP6 or 4 sockaddr and return the address
fs/dlm/lowcomms-tcp.c line 203
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+length */ +static void make_sockaddr (struct sockaddr_storage *saddr,uint16_t port, +int *addr_len) +{ + saddr->ss_family = dlm_local_addr.ss_family; +if (saddr->ss_family ==AF_INET ) { +struct sockaddr_in *in4_addr = (struct sockaddr_in *)saddr; + in4_addr->sin_port =cpu_to_be16 (port); + *addr_len =sizeof (struct sockaddr_in ); + }
whitespace broke
Review
Quoted 1 line
(quoted diff) +1
-0+ else {
} else {fs/dlm/lowcomms-sctp.c
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+struct sockaddr_in6 *in6_addr = (struct sockaddr_in6 *)saddr; + in6_addr->sin6_port =cpu_to_be16 (port); + *addr_len =sizeof (struct sockaddr_in6 ); + } +} + +static struct socket *create_listen_sock (struct connection *con,struct sockaddr_storage *saddr)
Exceeds 80 cols (lots of places)
fs/dlm/lowcomms-tcp.c line 613
Quoted 3 lines
fs/dlm/lowcomms-tcp.c new file +3
-0+{ +struct socket *sock =NULL ; +mm_segment_t fs;
whitespace broke
Quoted 59 lines
fs/dlm/lowcomms-tcp.c new file +59
-0+int result =0 ; +int one =1 ; +int addr_len; + +if (dlm_local_addr.ss_family ==AF_INET ) + addr_len =sizeof (struct sockaddr_in ); +else + addr_len =sizeof (struct sockaddr_in6 ); + +/* Create a socket to communicate with */ + result =sock_create_kern (dlm_local_addr.ss_family,SOCK_STREAM ,IPPROTO_TCP , &sock); +if (result <0 ) { +printk ("dlm: Can't create listening comms socket\n" ); + goto create_out; + } + + fs =get_fs (); +set_fs (get_ds ()); + result =sock_setsockopt (sock,SOL_SOCKET ,SO_REUSEADDR , (char *)&one,sizeof (one)); +set_fs (fs); +if (result <0 ) { +printk ("dlm: Failed to set SO_REUSEADDR on socket: result=%d\n" ,result); + } + sock->sk->sk_user_data = con; + con->rx_action = accept_from_sock; + con->sock = sock; + +/* Bind to our port */ +make_sockaddr (saddr, dlm_config.tcp_port, &addr_len); + result = sock->ops->bind (sock, (struct sockaddr *) saddr, addr_len); +if (result <0 ) { +printk ("dlm: Can't bind to port %d\n" , dlm_config.tcp_port); +sock_release (sock); + sock =NULL ; + con->sock =NULL ; + goto create_out; + } + + fs =get_fs (); +set_fs (get_ds ()); + + result =sock_setsockopt (sock,SOL_SOCKET ,SO_KEEPALIVE , (char *)&one,sizeof (one)); +set_fs (fs); +if (result <0 ) { +printk ("dlm: Set keepalive failed: %d\n" , result); + } + + result = sock->ops->listen (sock,5 ); +if (result <0 ) { +printk ("dlm: Can't listen on port %d\n" , dlm_config.tcp_port); +sock_release (sock); + sock =NULL ; + goto create_out; + } + + create_out: +return sock; +} +
fs/dlm/lowcomms-tcp.c line 682
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+ +/* Listen on all interfaces */ +static int listen_for_all (void ) +{ +struct socket *sock =NULL ; +struct connection *con =nodeid2con (0 ,GFP_KERNEL ); +int result = -EINVAL ; + +/* We don't support multi-homed hosts */ +memset (con,0 ,sizeof (*con));
It would be more efficient to make nodeid2con() return a zeroed object.
Quoted 30 lines
fs/dlm/lowcomms-tcp.c new file +27
-0+init_rwsem (&con->sock_sem); +spin_lock_init (&con->writequeue_lock); +INIT_LIST_HEAD (&con->writequeue); +set_bit (CF_IS_OTHERCON , &con->flags); + + sock =create_listen_sock (con, &dlm_local_addr); +if (sock) { +add_sock (sock, con); + result =0 ; + } +else { + result = -EADDRINUSE ; + } + +return result; +} + +} + +void *dlm_lowcomms_get_buffer (int nodeid,int len, +gfp_t allocation,char **ppc) +{ +struct connection *con; +struct writequeue_entry *e; +int offset =0 ; +int users =0 ; +
fs/dlm/lowcomms-tcp.c line 743
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+if (!atomic_read (&accepting)) +return NULL ; + + con =nodeid2con (nodeid, allocation); +if (!con) +return NULL ; + +spin_lock (&con->writequeue_lock); + e = list_entry(con->writequeue.prev, struct writequeue_entry, list); + if (((struct list_head *) e == &con->writequeue) ||
There's that cast again
fs/dlm/lowcomms-tcp.c line 756
Quoted 13 lines
fs/dlm/lowcomms-tcp.c new file +13
-0+ (PAGE_CACHE_SIZE - e->end < len)) { + e = NULL; + } else { + offset = e->end; + e->end += len; + users = e->users++; + } +spin_unlock (&con->writequeue_lock); + +if (e) { + got_one: +if (users ==0 ) +kmap (e->page);
Please check that this kmap (and the others) don't get leaked.
Quoted 46 lines
fs/dlm/lowcomms-tcp.c new file +46
-0+ *ppc =page_address (e->page) + offset; +return e; + } + + e =new_writequeue_entry (con, allocation); +if (e) { +spin_lock (&con->writequeue_lock); + offset = e->end; + e->end += len; + users = e->users++; +list_add_tail (&e->list, &con->writequeue); +spin_unlock (&con->writequeue_lock); + goto got_one; + } +return NULL ; +} + +/* Send a message */ +static int send_to_sock (struct connection *con) +{ +int ret =0 ; +ssize_t (*sendpage) (struct socket *,struct page *,int ,size_t ,int ); +const int msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ; +struct writequeue_entry *e; +int len, offset; + +down_read (&con->sock_sem); +if (con->sock ==NULL ) + goto out_connect; + + sendpage = con->sock->ops->sendpage; + +spin_lock (&con->writequeue_lock); +for (;;) { + e = list_entry(con->writequeue.next, struct writequeue_entry, + list); +if ((struct list_head *) e == &con->writequeue) +break ; + + len = e->len; + offset = e->offset; +BUG_ON (len ==0 && e->users ==0 ); +spin_unlock (&con->writequeue_lock); + + ret =0 ; +if (len) {
fs/dlm/lowcomms-tcp.c line 849
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+ ret =sendpage (con->sock, e->page, offset, len, + msg_flags); +if (ret == -EAGAIN || ret ==0 ) + goto out; +if (ret <=0 ) + goto send_error; + } +else { +/* Don't starve people filling buffers */ +schedule ();
cond_resched()
Quoted 15 lines
fs/dlm/lowcomms-tcp.c new file +15
-0+ } + +spin_lock (&con->writequeue_lock); + e->offset += ret; + e->len -= ret; + +if (e->len ==0 && e->users ==0 ) { +list_del (&e->list); +free_entry (e); +continue ; + } + } +spin_unlock (&con->writequeue_lock); + out: +up_read (&con->sock_sem);
fs/dlm/lowcomms-tcp.c line 874
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+return ret; + + send_error: +up_read (&con->sock_sem); +close_connection (con, FALSE); +lowcomms_connect_sock (con); +return ret; + + out_connect: +up_read (&con->sock_sem);
More wayward labels there.
Quoted 17 lines
fs/dlm/lowcomms-tcp.c new file +14
-0+lowcomms_connect_sock (con); +return 0 ; +} + +/* Try to send any messages that are pending +*/ +static void process_output_queue (void ) +{ +struct list_head *list; +struct list_head *temp; +int ret; + +spin_lock_bh (&write_sockets_lock); +list_for_each_safe (list, temp, &write_sockets) {
fs/dlm/lowcomms-tcp.c line 993
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+struct connection *con = + list_entry(list, struct connection, write_list); +clear_bit (CF_WRITE_PENDING , &con->flags); +list_del (&con->write_list); + +spin_unlock_bh (&write_sockets_lock); + + ret =send_to_sock (con); +if (ret <0 ) { + }
hmm.
Quoted 12 lines
fs/dlm/lowcomms-tcp.c new file +12
-0+spin_lock_bh (&write_sockets_lock); + } +spin_unlock_bh (&write_sockets_lock); +} + +static void process_state_queue (void ) +{ +struct list_head *list; +struct list_head *temp; +int ret; + +spin_lock_bh (&state_sockets_lock);
fs/dlm/lowcomms-tcp.c line 1015
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+list_for_each_safe (list, temp, &state_sockets) { +struct connection *con = + list_entry(list, struct connection, state_list); +list_del (&con->state_list); +clear_bit (CF_CONNECT_PENDING , &con->flags); +spin_unlock_bh (&state_sockets_lock); + + ret =connect_to_sock (con); +if (ret <0 ) { + }
?
Quoted 19 lines
fs/dlm/lowcomms-tcp.c new file +19
-0+spin_lock_bh (&state_sockets_lock); + } +spin_unlock_bh (&state_sockets_lock); +} + + +/* Discard all entries on the write queues */ +static void clean_writequeues (void ) +{ +int nodeid; + +for (nodeid =1 ; nodeid < conn_array_size; nodeid++) { +struct connection *con =nodeid2con (nodeid,0 ); + +if (con) +clean_one_writequeue (con); + } +} +
fs/dlm/lowcomms-tcp.c line 1044
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+static int read_list_empty (void ) +{ +int status; + +spin_lock_bh (&read_sockets_lock); + status =list_empty (&read_sockets); +spin_unlock_bh (&read_sockets_lock); + +return status; +}
Again, unsafe.
Quoted 9 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+static int daemons_start (void ) +{ +struct task_struct *p; +int error; + + p =kthread_run (dlm_recvd,NULL ,"dlm_recvd" ); + error =IS_ERR (p); +if (error) { +log_print ("can't start dlm_recvd %d" , error);
fs/dlm/lowcomms-sctp.c line 1180
Quoted 10 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+return error; + } + recv_task = p; + + p =kthread_run (dlm_sendd,NULL ,"dlm_sendd" ); + error =IS_ERR (p); +if (error) { +log_print ("can't start dlm_sendd %d" , error); +kthread_stop (recv_task); +return error;
whitespace broke.
Quoted 21 lines
fs/dlm/lowcomms-tcp.c new file +18
-0+ } + send_task = p; + +return 0 ; +} + +/* This is quite likely to sleep... */ +int dlm_lowcomms_start (void ) +{ +int error =0 ; + + error = -ENOTCONN ; + +/* +* Temporarily initialise the waitq head so that lowcomms_send_message +* doesn't crash if it gets called before the thread is fully +* initialised +*/
fs/dlm/lowcomms-tcp.c line 1192
Quoted 10 lines
fs/dlm/lowcomms-tcp.c new file +10
-0+init_waitqueue_head (&lowcomms_send_waitq); + + error = -ENOMEM ; + connections =kmalloc (sizeof (struct connection *) * +NODE_INCREMENT ,GFP_KERNEL ); +if (!connections) + goto out; + +memset (connections,0 , +sizeof (struct connection *) *NODE_INCREMENT );
more kzalloc()
Quoted 42 lines
fs/dlm/lowcomms-tcp.c new file +42
-0+ conn_array_size =NODE_INCREMENT ; + +if (dlm_our_addr (&dlm_local_addr,0 )) { +log_print ("no local IP address has been set" ); + goto fail_free_conn; + } +if (!dlm_our_addr (&dlm_local_addr,1 )) { +log_print ("This dlm comms module does not support multi-homed clustering" ); + goto fail_free_conn; + } + + con_cache =kmem_cache_create ("dlm_conn" ,sizeof (struct connection ), + __alignof__(struct connection ),0 ,NULL ,NULL ); +if (!con_cache) + goto fail_free_conn; + + +/* Start listening */ + error =listen_for_all (); +if (error) + goto fail_unlisten; + + error =daemons_start (); +if (error) + goto fail_unlisten; + +atomic_set (&accepting,1 ); + +return 0 ; + + fail_unlisten: +close_connection (connections[0 ],0 ); +kmem_cache_free (con_cache, connections[0 ]); +kmem_cache_destroy (con_cache); + + fail_free_conn: +kfree (connections); + + out: +return error; +} +
Attached is a patch to fix up most of the things pointed out by akpm and Pavel Machek with comments below indicating why some things have been left.
The patch is against -nmw
Quoted 11 lines · Andrew Morton
fs/dlm/lowcomms-sctp.c line 183
Quoted 8 lines
fs/dlm/lowcomms-sctp.c new file +7
-0+static struct nodeinfo *nodeid2nodeinfo (int nodeid,gfp_t alloc) +{ +struct nodeinfo *ni; +int r; +int n; + +down_read (&nodeinfo_lock);Given that this function can sleep, I wonder if `alloc' is useful.
Quoted 7 lines
I see lots of callers passing in a literal "0" for `alloc'. That's in fact a secret (
GFP_ATOMIC& ~__GFP_HIGH). I doubt if that's what you really meant. Particularly as the code could at least have used__GFP_WAIT(akaGFP_NOIO) which is much, much more reliable than "0". In fact "0" is the least reliable mode possible.IOW, this is all bollixed up.
When 0 is passed into nodeid2nodeinfo the function does not try to allocate a new structure at all. it's an indication that the caller only wants the nodeinfo struct for that nodeid if there actually is one in existance.
I've tidied the function itself so it's more obvious, (and tidier!)
Quoted 12 lines
Quoted 12 lines
fs/dlm/lowcomms-sctp.c new file +12
-0+/* Data received from remote end */ +static int receive_from_sock (void ) +{ +int ret =0 ; +struct msghdr msg; +struct kvec iov[2 ]; +unsigned len; +int r; +struct sctp_sndrcvinfo *sinfo; +struct cmsghdr *cmsg; +struct nodeinfo *ni; +
Quoted 8 lines
fs/dlm/lowcomms-sctp.c line 497
Quoted 6 lines
fs/dlm/lowcomms-sctp.c new file +6
-0+/* These two are marginally too big for stack allocation, but this +* function is (currently) only called by dlm_recvd so static should be +* OK. +*/ +static struct sockaddr_storage msgname; +static char incmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];whoa. This is globally singly-threaded code??
Yes. it is only ever run in the context of dlm_recvd.
Quoted 7 lines
fs/dlm/lowcomms-sctp.c line 843
Quoted 5 lines
fs/dlm/lowcomms-sctp.c new file +4
-0+static void initiate_association (int nodeid) +{ +struct sockaddr_storage rem_addr; +static char outcmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];Another static buffer to worry about. Globally singly-threaded code?
Yes. Only ever called by dlm_sendd.
Quoted 11 lines
fs/dlm/lowcomms-sctp.c line 901
Quoted 9 lines
fs/dlm/lowcomms-sctp.c new file +9
-0+ +/* Send a message */ +static int send_to_sock (struct nodeinfo *ni) +{ +int ret =0 ; +struct writequeue_entry *e; +int len, offset; +struct msghdr outmsg; +static char outcmsg[CMSG_SPACE (sizeof (struct sctp_sndrcvinfo ))];Singly-threaded?
Yep.
Quoted 11 lines
fs/dlm/lowcomms-sctp.c line 1061
Quoted 9 lines
fs/dlm/lowcomms-sctp.c new file +8
-0+static void dealloc_nodeinfo (void ) +{ +int i; + +for (i=1 ; i<=max_nodeid; i++) { +struct nodeinfo *ni =nodeid2nodeinfo (i,0 ); +if (ni) { +idr_remove (&nodeinfo_idr, i);Didn't that need locking?
Not. it's only ever called at DLM shutdown after all the other threads have been stopped.
Quoted 15 lines
fs/dlm/lowcomms-sctp.c line 1097
Quoted 11 lines
fs/dlm/lowcomms-sctp.c new file +10
-0+static int write_list_empty (void ) +{ +int status; + +spin_lock_bh (&write_nodes_lock); + status =list_empty (&write_nodes); +spin_unlock_bh (&write_nodes_lock); + +return status; +}This function's return value is meaningless. As soon as the lock gets dropped, the return value can get out of sync with reality.
Quoted 2 lines
Looking at the caller, this _might_ happen to be OK, but it's a nasty and dangerous thing. Really the locking should be moved into the caller.
It's just an optimisation to allow the caller to schedule if there is no work to do. if something arrives immediately afterwards then it will get picked up when the process re-awakes (and it will be woken by that arrival).
The 'accepting' atomic has gone completely. as Andrew pointed out it didn't really achieve much anyway. I suspect it was a plaster over some other startup or shutdown bu to be honest.
patrick
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: lowcomms.tidy
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20061206/bc872d11/attachment.ksh>