Summary of 2005 Kernel Summit Proposed Topics
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 1111628393.1548.307.camel@beastie
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 1111633846.1548.318.camel@beastie
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
I'm cc:ing this to netdev, where this discussion really ought to be. There's a separate networking summit and I suspect most of the networking heavies aren't reading ksummit-discuss or open-iscsi. It's getting rather far afield for ksummit-discuss so people should trim that from follow-ups.
Quoted 7 lines · Andrea Arcangeli
Quoted 3 lines · Matt Mackall
There may be network multipath. But I think we can have a single socket mempool per logical device and a single skbuff mempool shared among those sockets.
If we'll have to reserve more than 1 packet per each socket context, then the mempool probably can't be shared.
I believe the mempool can be shared among all sockets that represent the same storage device. Packets out any socket represent progress.
Quoted 2 lines
I wonder if somebody has ever reproduced deadlocks by swapping on software-tcp-iscsi.
Yes, done before it was even called iSCSI.
Quoted 4 lines
Quoted 2 lines
And that still leaves us with the lack of buffers to receive ACKs problem, which is perhaps worse.
The mempooling should take care of the acks too.
The receive buffer is allocated at the time we DMA it from the card. We have no idea of its contents and we won't know what socket mempool to pull the receive skbuff from until much higher in the network stack, which could be quite a while later if we're under OOM load. And we can't have a mempool big enough to handle all the traffic that might potentially be deferred for softirq processing when we're OOM, especially at gigabit rates.
I think this is actually the tricky piece of the problem and solving the socket and send buffer allocation doesn't help until this gets figured out.
We could perhaps try to address this with another special receive-side alloc_skb that fails most of the time on OOM but sometimes pulls from a special reserve.
Quoted 4 lines
Perhaps the mempooling overhead will be too huge to pay for it even when it's not necessary, in such case the iscsid will have to pass a new bitflag to the socket syscall, when it creates the socket meant to talk with the remote disk.
I think we probably attach a mempool to a socket after the fact. And no, we can't have a mempool attached to every socket.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 2 lines · Matt Mackall
I believe the mempool can be shared among all sockets that represent the same storage device. Packets out any socket represent progress.
What's the point to have more than one socket connected to each storage device anyway?
Quoted 1 line
Yes, done before it was even called iSCSI.
Ok, theoretical deadlock conditions aren't nice anyway, but knowing this is a real life problem too makes it more interesting ;).
Quoted 8 lines
The receive buffer is allocated at the time we DMA it from the card. We have no idea of its contents and we won't know what socket mempool to pull the receive skbuff from until much higher in the network stack, which could be quite a while later if we're under OOM load. And we can't have a mempool big enough to handle all the traffic that might potentially be deferred for softirq processing when we're OOM, especially at gigabit rates.
Quoted 7 lines
I think this is actually the tricky piece of the problem and solving the socket and send buffer allocation doesn't help until this gets figured out.
We could perhaps try to address this with another special receive-side
alloc_skbthat fails most of the time on OOM but sometimes pulls from a special reserve.
One algo to handle this is: after we get the gfp_atomic failure, we look at all the mempools are registered for a certain NIC, and we pick a random mempools that isn't empty. We use the non-empty mempool to receive the packet, and we let the netif_rx process the packet. Then if going up the stack we find that the packet doesn't belong to the socket-mempool, we discard the packet and we release the ram back into the mempool. This should make progress since eventually the right packet will go in the right mempool.
Quoted 6 lines
Quoted 4 lines
Perhaps the mempooling overhead will be too huge to pay for it even when it's not necessary, in such case the iscsid will have to pass a new bitflag to the socket syscall, when it creates the socket meant to talk with the remote disk.
I think we probably attach a mempool to a socket after the fact. And
I guess you meant before the fact (i.e. before the connection to the server), anything attached after the fact (whatever the fact is ;) isn't going to help.
Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines · Matt Mackall
I believe the mempool can be shared among all sockets that represent the same storage device. Packets out any socket represent progress.
What's the point to have more than one socket connected to each storage device anyway?
There may be multiple network addresses (with different network paths) associated with the same device for purposes of throughput or reliability.
Quoted 8 lines
One algo to handle this is: after we get the
gfp_atomicfailure, we look at all the mempools are registered for a certain NIC, and we pick a random mempools that isn't empty. We use the non-empty mempool to receive the packet, and we let thenetif_rxprocess the packet. Then if going up the stack we find that the packet doesn't belong to the socket-mempool, we discard the packet and we release the ram back into the mempool. This should make progress since eventually the right packet will go in the right mempool.
What if the number of packets queued by the time we reach the softirq side of the stack exceeds the available buffers?
Imagine that we've got heavy DNS and iSCSI on the same box and that the box gets wedged in OOM such that it can't answer DNS queries. But we can't distinguish at receive time between DNS and iSCSI. As iSCSI is TCP, it will send repeat ACKs at relatively long intervals but the DNS clients will potentially continue to hammer the machine, filling the reserve buffers and starving out the ACKs. We've got to essentially be able to say "we are OOM, drop all traffic to sockets not flagged for storage" and do so quickly enough that we can eventually get the ACKs.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Perhaps the mempooling overhead will be too huge to pay for it even when it's not necessary, in such case the iscsid will have to pass a new bitflag to the socket syscall, when it creates the socket meant to talk with the remote disk.
I think we probably attach a mempool to a socket after the fact. And
I guess you meant before the fact (i.e. before the connection to the server), anything attached after the fact (whatever the fact is ;) isn't going to help.
After the socket is created, but before we commit to pumping storage data through it (iSCSI has multiple phases). A privileged setsockopt-like interface ought to suffice. Or something completely kernel internal.
Which reminds me: FUSE and friends presumably have a very similar set of problems.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 2 lines · Matt Mackall
What if the number of packets queued by the time we reach the softirq side of the stack exceeds the available buffers?
That means they weren't for the iscsi socket and they will be discarded right away (instead of queueing them in the sock).
Quoted 3 lines
Imagine that we've got heavy DNS and iSCSI on the same box and that the box gets wedged in OOM such that it can't answer DNS queries. But we can't distinguish at receive time between DNS and iSCSI. As iSCSI is TCP, it
We don't care about performance here, if we're under a flood attack it'll take a long time but as long as you keep discarding them right away as soon as you notice the reservation wasn't for the current sock, it should keep making progress and not deadlock anymore.
This is a deadlock vs non-deadlock issue, how fast the other packets arrives is a secondary issue, we're in a slow path.
Quoted 3 lines
will send repeat ACKs at relatively long intervals but the DNS clients will potentially continue to hammer the machine, filling the reserve buffers and starving out the ACKs. We've got to essentially be able to
They won't emtpy it, since they will be released immediatly. From the ack standpoint it'll be like packet loss due network congestion, infact this sounds close to network congestion.
Quoted 2 lines
say "we are OOM, drop all traffic to sockets not flagged for storage" and do so quickly enough that we can eventually get the ACKs.
To do that you've to reserve a NIC for that. But the whole point of the algo I proposed is to work fine with shared NIC to avoid the deadlock too (it won't resolve it in a high performant way, but the issue is that it won't be a deadlock condition anymore). And if the reserved buffer is huge likely you won't lose many packets at all.
Quoted 36 lines · Matt Mackall
I'm cc:ing this to netdev, where this discussion really ought to be. There's a separate networking summit and I suspect most of the networking heavies aren't reading ksummit-discuss or open-iscsi. It's getting rather far afield for ksummit-discuss so people should trim that from follow-ups.
Quoted 7 lines · Andrea Arcangeli
Quoted 3 lines · Matt Mackall
There may be network multipath. But I think we can have a single socket mempool per logical device and a single skbuff mempool shared among those sockets.
If we'll have to reserve more than 1 packet per each socket context, then the mempool probably can't be shared.
I believe the mempool can be shared among all sockets that represent the same storage device. Packets out any socket represent progress.
Quoted 2 lines
I wonder if somebody has ever reproduced deadlocks by swapping on software-tcp-iscsi.
Yes, done before it was even called iSCSI.
Quoted 4 lines
Quoted 2 lines
And that still leaves us with the lack of buffers to receive ACKs problem, which is perhaps worse.
The mempooling should take care of the acks too.
The receive buffer is allocated at the time we DMA it from the card. We have no idea of its contents and we won't know what socket mempool to pull the receive skbuff from until much higher in the network stack, which could be quite a while later if we're under OOM load. And we can't have a mempool big enough to handle all the traffic that might potentially be deferred for softirq processing when we're OOM, especially at gigabit rates.
Quoted 7 lines
I think this is actually the tricky piece of the problem and solving the socket and send buffer allocation doesn't help until this gets figured out.
We could perhaps try to address this with another special receive-side
alloc_skbthat fails most of the time on OOM but sometimes pulls from a special reserve.
nope. this will not solve the problem on receive or will just solve it partially. The right way to solve it would be to provide special API which will help to re-use link-layer's ring SKBs. i.e. TCP stack should call NIC driver's callback after all SKB data been successfully copied to the user space. At that point NIC driver will safely replenish HW ring. This way we could avoid most of memory allocations on receive.
Quoted 8 lines
Quoted 4 lines
Perhaps the mempooling overhead will be too huge to pay for it even when it's not necessary, in such case the iscsid will have to pass a new bitflag to the socket syscall, when it creates the socket meant to talk with the remote disk.
I think we probably attach a mempool to a socket after the fact. And no, we can't have a mempool attached to every socket.
David S. Miller
maintainerQuoted 4 lines · On Sat, 26 Mar 2005 22:33:01 -0800 Dmitry Yusupov <[email protected]> wrote:
i.e. TCP stack should call NIC driver's callback after all SKB data been successfully copied to the user space. At that point NIC driver will safely replenish HW ring. This way we could avoid most of memory allocations on receive.
How does this solve your problem? This is just simple SKB recycling, and it's a pretty old idea.
TCP packets can be held on receive for arbitrary amounts of time.
This is especially true if data is received out of order or when packets are dropped. We can't even wake up the user until the holes in the sequence space are filled.
Even if data is received properly and in order, there are no hard guarentees about when the user will get back onto the CPU to get the data copied to it.
During these gaps in time, you will need to keep your HW receive ring populated with packets.
Quoted 10 lines · David S. Miller
Quoted 4 lines · On Sat, 26 Mar 2005 22:33:01 -0800 Dmitry Yusupov <[email protected]> wrote:
i.e. TCP stack should call NIC driver's callback after all SKB data been successfully copied to the user space. At that point NIC driver will safely replenish HW ring. This way we could avoid most of memory allocations on receive.
How does this solve your problem? This is just simple SKB recycling, and it's a pretty old idea.
I know. it is very old idea.
Quoted 1 line
TCP packets can be held on receive for arbitrary amounts of time.
I'm thinking about mixing existing way of doing things with guarantied SKB recycling. It should help to storage stacks to make a progress on receive at least.
Quoted 10 lines
This is especially true if data is received out of order or when packets are dropped. We can't even wake up the user until the holes in the sequence space are filled.
Even if data is received properly and in order, there are no hard guarentees about when the user will get back onto the CPU to get the data copied to it.
During these gaps in time, you will need to keep your HW receive ring populated with packets.
ethernet flow-control must take care this case.
If driver's replenish logic could mix alloc_skb/netif_rx and SKB recycling than pause frames should never happen even with gige+ interfaces.
David S. Miller
maintainerQuoted 8 lines · On Sat, 26 Mar 2005 23:05:30 -0800 Dmitry Yusupov <[email protected]> wrote:
Quoted 2 lines
During these gaps in time, you will need to keep your HW receive ring populated with packets.
ethernet flow-control must take care this case.
If driver's replenish logic could mix alloc_skb/netif_rx and SKB recycling than pause frames should never happen even with gige+ interfaces.
I don't see what the big deal is if pause frames are generated when the system is low on atomic memory and RX allocations thus fail.
SKB recycling doesn't get the user on the cpu faster to receive the data. I don't understand how you expect the recycling to be guarenteed except perhaps as a special case for iSCSI taking in the TCP packets in the ->data_ready() callback. In that case it's exactly that, a special case hack, and not something generically useful at all.
Quoted 8 lines · David S. Miller
Quoted 5 lines · On Sat, 26 Mar 2005 23:05:30 -0800 Dmitry Yusupov <[email protected]> wrote:
Quoted 2 lines
During these gaps in time, you will need to keep your HW receive ring populated with packets.
ethernet flow-control must take care this case.
Quoted 7 lines
Quoted 3 lines
If driver's replenish logic could mix alloc_skb/netif_rx and SKB recycling than pause frames should never happen even with gige+ interfaces.
I don't see what the big deal is if pause frames are generated when the system is low on atomic memory and RX allocations thus fail.
not a big deal may be. but. very interesting case when OOM causing paging in/out and swapping device are on the same network under iSCSI control. (disk-less setups) having reliable receive in that case is important for making progress for READ operations.
Quoted 6 lines
SKB recycling doesn't get the user on the cpu faster to receive the data. I don't understand how you expect the recycling to be guarenteed except perhaps as a special case for iSCSI taking in the TCP packets in the ->data_ready() callback. In that case it's exactly that, a special case hack, and not something generically useful at all.
right. this is what Open-iSCSI project is using for READs.
Quoted 20 lines · Dmitry Yusupov
Quoted 9 lines · David S. Miller
Quoted 5 lines · On Sat, 26 Mar 2005 23:05:30 -0800 Dmitry Yusupov <[email protected]> wrote:
Quoted 2 lines
During these gaps in time, you will need to keep your HW receive ring populated with packets.
ethernet flow-control must take care this case.
Quoted 7 lines
Quoted 3 lines
If driver's replenish logic could mix alloc_skb/netif_rx and SKB recycling than pause frames should never happen even with gige+ interfaces.
I don't see what the big deal is if pause frames are generated when the system is low on atomic memory and RX allocations thus fail.
Quoted 4 lines
not a big deal may be. but. very interesting case when OOM causing paging in/out and swapping device are on the same network under iSCSI control. (disk-less setups) having reliable receive in that case is important for making progress for READ operations.
reliable receive is ciritical for WRITEs. Even if the WRITE is executed successfully on the remote device, if we cannot receive the return status from the device the operation will fail at the iscsi driver side due to a SCSI timeout.
Quoted 12 lines
Quoted 8 lines
SKB recycling doesn't get the user on the cpu faster to receive the data. I don't understand how you expect the recycling to be guarenteed except perhaps as a special case for iSCSI taking in the TCP packets in the ->data_ready() callback. In that case it's exactly that, a special case hack, and not something generically useful at all.
right. this is what Open-iSCSI project is using for READs.
David S. Miller
maintainerQuoted 4 lines · On Sun, 27 Mar 2005 10:26:29 -0800 Mike Christie <[email protected]> wrote:
reliable receive is ciritical for WRITEs. Even if the WRITE is executed successfully on the remote device, if we cannot receive the return status from the device the operation will fail at the iscsi driver side due to a SCSI timeout.
I keep hearing this word "reliable", it means something very different for TCP over a transport like IP than it does for the SCSI layer.
It is, in fact, the whole difficulty of implementing iSCSI:
being able to cope with this difference in expectations.
All I can see is that the SCSI layer's timeout is inappropriate for something like iSCSI, not that TCP or networking needs to change in some way.
Quoted 11 lines · David S. Miller
Quoted 4 lines · On Sun, 27 Mar 2005 10:26:29 -0800 Mike Christie <[email protected]> wrote:
reliable receive is ciritical for WRITEs. Even if the WRITE is executed successfully on the remote device, if we cannot receive the return status from the device the operation will fail at the iscsi driver side due to a SCSI timeout.
I keep hearing this word "reliable", it means something very different for TCP over a transport like IP than it does for the SCSI layer.
This has nothing to do with the specifics of TCP or IP.
We are out of memory. To free memory, we must be able to send N packets and receive M acknowledgements. Sending and receiving packets requires allocations - if we cannot allocate, we are permanently wedged.
This strongly suggests we need to have private reserves that the network layer knows how to access to fullfill higher level write requests. A closely analogous situation exists with regular SCSI, where the transport is also potentially lossy.
(In the iSCSI case, it's somewhat worse: we may need to open a new socket or have one in reserve.)
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 8 lines · David S. Miller
Quoted 4 lines · On Sun, 27 Mar 2005 10:26:29 -0800 Mike Christie <[email protected]> wrote:
reliable receive is ciritical for WRITEs. Even if the WRITE is executed successfully on the remote device, if we cannot receive the return status from the device the operation will fail at the iscsi driver side due to a SCSI timeout.
Quoted 6 lines
I keep hearing this word "reliable", it means something very different for TCP over a transport like IP than it does for the SCSI layer.
It is, in fact, the whole difficulty of implementing iSCSI:
being able to cope with this difference in expectations.
actually, you absolutely right!
there are two ways to achieve reasonable "reliability" of iSCSI transport:
1) ERL=2(see RFC3720) + multiple connections (read: single iSCSI session with multiple connections)
2) ERL=0 + device multipath (read: multiple iSCSI sessions with single connection per session)
I think we should slow down on "reliability" feature since 1) and 2) basically covers the case. All we need is to make sure that deadlock situations during OOM will never happen.
Quoted 3 lines
All I can see is that the SCSI layer's timeout is inappropriate for something like iSCSI, not that TCP or networking needs to change in some way.
right. looks like there is the way to "postpone" SCSI timeout until iSCSI session will finally recover.
Quoted 26 lines · Mike Christie
Quoted 20 lines · Dmitry Yusupov
Quoted 9 lines · David S. Miller
Quoted 5 lines · On Sat, 26 Mar 2005 23:05:30 -0800 Dmitry Yusupov <[email protected]> wrote:
Quoted 2 lines
During these gaps in time, you will need to keep your HW receive ring populated with packets.
ethernet flow-control must take care this case.
Quoted 7 lines
Quoted 3 lines
If driver's replenish logic could mix alloc_skb/netif_rx and SKB recycling than pause frames should never happen even with gige+ interfaces.
I don't see what the big deal is if pause frames are generated when the system is low on atomic memory and RX allocations thus fail.
Quoted 4 lines
not a big deal may be. but. very interesting case when OOM causing paging in/out and swapping device are on the same network under iSCSI control. (disk-less setups) having reliable receive in that case is important for making progress for READ operations.
Quoted 4 lines
reliable receive is ciritical for WRITEs. Even if the WRITE is executed successfully on the remote device, if we cannot receive the return status from the device the operation will fail at the iscsi driver side due to a SCSI timeout.
of course. forget to mention this at the first place. WRITE needs successful SCSI response too. until that it will not be returned back to the SCSI Mid-Layer.
Quoted 20 lines · David S. Miller
Quoted 2 lines · On Sat, 26 Mar 2005 22:33:01 -0800 Dmitry Yusupov <[email protected]> wrote:
i.e. TCP stack should call NIC driver's callback after all SKB data been successfully copied to the user space. At that point
NIC driver
Quoted 1 line
will safely replenish HW ring. This way we could avoid most
of memory
Quoted 1 line
allocations on receive.
How does this solve your problem? This is just simple SKB recycling, and it's a pretty old idea.
TCP packets can be held on receive for arbitrary amounts of time.
This is especially true if data is received out of order or when packets are dropped. We can't even wake up the user until the holes in the sequence space are filled.
Quoted 6 lines
Even if data is received properly and in order, there are no hard guarentees about when the user will get back onto the CPU to get the data copied to it.
During these gaps in time, you will need to keep your HW receive ring populated with packets.
Here's the way I see it.
1) There are iSCSI connections that should be "protected", resources-wise.
Examples: remote swap device, bank accounts database on RAID accessed via iSCSI, etc.
2) There are two ways to protect the "protected" connections. One "Big Brother" like way is a centralized Resource Manager that performs a fully deterministic resource accounting throughout the system, all the way from NIC descriptors and on-chip memory up to iSCSI buffers for Data-Out headers.
3) The 2nd way is *awareness* of the "protected" connections propagated throughout the system, along with incremental implementation of more sophisticated recovery schemes.
4) The Resource Manager could be used in the following way. At session open time iSCSI control plane calculates iSCSI and TCP resources that should be available at all times. The calculation is done based on: the number of SCSI commands to be processed in parallel (the 'can_queue'), the maximum size of the SCSI payload in the SG, the negotiated maximum number of outstanding R2Ts, sizes of Immediate and FirstBurst data.
5) If Resource manager says there is not enough resources, iSCSI fails session open. This is better than to get in trouble well into runtime.
6) For example: to transmit 'can_queue' commands, iSCSI needs N skbufs. Let's say, all N commands transmitted in a burst, and just one of these N gets ack-ed by the Target (via StatSN). In the fully deterministic system this does not necessarily mean that the scsi-ml can now send one command - because the full condition involves also recycling of skbuf(s) used for transmitting this one completed command. And although it is hard to imagine that the command gets fully done by the remote target without Tx buffers getting recycled, the theoretical chance exists (e.g., the NIC is slow or the driver has a bad Tx recycling implementation), and the fully deterministic scheme should take it into account.
7) Therefore, prior to calling scsi_done() iSCSI asks Resource Manager whether all the TCP etc. resources used for this command are already recycled. If not, the scsi_done() gets postponed. In addition, iSCSI "complains" to Resource Manager that it enters slow path because of this, which could prompt the latter to take an action. (End of the example).
8) If we agree to declare some connections "resource-proteced", it would immediately mean that there are possibly other connections that are not (resource-protected). Which in turn gives the Resource Manager a flexibility to OOM-kill those unprotected connections and cannibalize the corresponding resources for the protected ones.
9) Without some awareness of the resource-protected connections, and without some kind of resource counting at runtime (let it be partial and incomplete for starters) - the only remaining way for customers that require HA (High Availability) is to over-engineer: use 64GB RAM, TBs of disk space, etc. Which is probably not the end of the world as long as the prices go down..
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Alex Aizman
5) If Resource manager says there is not enough resources, iSCSI fails session open. This is better than to get in trouble well into runtime.
Yes, this is the concept we were calling mempooling in earlier emails:
reserve the ram during session open and abort before starting up if we fail. The kernel already does this in all I/O places.
Quoted 4 lines
9) Without some awareness of the resource-protected connections, and without some kind of resource counting at runtime (let it be partial and incomplete for starters) - the only remaining way for customers that require HA (High Availability) is to over-engineer: use 64GB RAM, TBs of disk space, etc.
... and most important set freepages.min to 1G or so (assuming 64bit arch of course). It's _only_ freepages.min that matters, the total ram doesn't matter at all, since you can mark it all dirty in a few seconds with MAP_SHARED and then it will be lost (i.e. unfreeable) until iscsi can succeed the write.
Quoted 4 lines · David S. Miller
How does this solve your problem? This is just simple SKB recycling, and it's a pretty old idea.
TCP packets can be held on receive for arbitrary amounts of time.
We shouldn't do that when we're really really low on memory. I envision something like this:
1) have iSCSI, NFS, etc. open their sockets with a socket option that indicates this is a VM deadlock sensitive socket (SO_MEMALLOC?) - these sockets get two mempools, one for sending and one for receiving
2) have a global emergency mempool available to receive network packets when GFP_ATOMIC fails - this is useful since we don't know who a packet is for when we get the NIC interrupt, and it's easy to have just one pool to check
3) when a packet is received through this mempool, check whether the packet is for an SO_MEMALLOC socket
==> if not, discard the packet, free the memory
4) if the packet is for an SO_MEMALLOC socket, and that socket has space left in its own receiving mempool, and the packet is not out of order, then transfer the data to the socket
==> at this point, the space in the global network receive mempool can be freed again
5) if we cannot handle the packet, drop it
This way:
1) memory critical sockets are protected from other network traffic
2) memory critical sockets are protected from each other
3) all memory critical sockets should be able to make progress
The only thing left would be the memory needed to do reconnects to the storage device, while in this situation. I suspect we may be able to keep that out of the network layerr, if we allow drivers like iSCSI to pass in their own memory reserve to get the mempool populated.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
David S. Miller
maintainerQuoted 2 lines · On Sun, 27 Mar 2005 22:54:11 -0500 (EST) Rik van Riel <[email protected]> wrote:
We shouldn't do that when we're really really low on memory. I envision something like this:
...
Quoted 1 line
(SO_MEMALLOC?)
Interesting top-down scheme.
We could also make a way to adjust the GFP_ATOMIC reserve thresholds too. That seems a bit more generic.
Quoted 9 lines · David S. Miller
Quoted 1 line · On Sun, 27 Mar 2005 22:54:11 -0500 (EST) Rik van Riel <[email protected]> wrote:
(SO_MEMALLOC?)Interesting top-down scheme.
We could also make a way to adjust the
GFP_ATOMICreserve thresholds too. That seems a bit more generic.
That might work, as long as we can guarantee that each SO_MEMALLOC socket has at least N amount of memory available just for itself, so none of these sockets get to deadlock.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 11 lines · Rik van Riel
1) have iSCSI, NFS, etc. open their sockets with a socket option that indicates this is a VM deadlock sensitive socket (
SO_MEMALLOC?) - these sockets get two mempools, one for sending and one for receiving
2) have a global emergency mempool available to receive network packets whenGFP_ATOMICfails - this is useful since we don't know who a packet is for when we get the NIC interrupt, and it's easy to have just one pool to check
3) when a packet is received through this mempool, check whether the packet is for anSO_MEMALLOCsocket
Quoted 9 lines
==> if not, discard the packet, free the memory
4) if the packet is for anSO_MEMALLOCsocket, and that socket has space left in its own receiving mempool, and the packet is not out of order, then transfer the data to the socket
==> at this point, the space in the global network receive mempool can be freed again
5) if we cannot handle the packet, drop it
Let's say, there are only iSCSI and NFS sockets (no UDP, which is some relief :-), and each opened with SO_MEMALLOC. The sockets are allowed to oversubscribe (via tcp_rmem, tcp_wmem etc. defaults and socket options), which means: the total amount of mempools memory can get beyond physically available. It often does, actually. Which means: the 5) on the list above. Once we allow for the non-determinism of an occasional packet drop, there's a chance for the retransmission to not go through.
Otherwise, it's a great incremental step. The stack (for starters, and ingoring for now resources of the NIC, iSCSI itself, etc.) needs to be aware that some connections are more important than others. Some are "resource-protected", others not. This is a useful piece of information.
Alex
Quoted 5 lines · Rik van Riel
one for sending and one for receiving
2) have a global emergency mempool available to receive network packets whenGFP_ATOMICfails - this is useful since we don't know who a packet is for when we get the NIC interrupt, and it's easy to have just one pool to check
This does not work because mempools assume you can sleep, and in most NIC drivers you cant while doing RX refill. The NIC drivers can be rewritten to do this refilling in a workqueue. But it is not clear it is useful anyways because Linux failing to allocate a buffer is no different from the network overflowing the hardware queue of the network device, which Linux cannot do anything about.
Basically a network consists of lots of interconnected queues, and even if you try to make the Linux specific side of the queue reliable there are lots of other queues that can still lose packets.
With TCP that is no problem of course because in case of a packet loss the packet is just retransmitted.
So in short using mempools on receiving is not needed.
Now memory allocation for writing is a different chapter. You cannot recover from a lost writing.
The kernel currently goes even in endless loops in this case (e.g. on TCP FIN allocation)
With the exception of routing the allocation fortunately usually happens there in:
- socket context: great you can use a per socket mempool
- thread context: you can sleep
With routing that is not the case, but it does not matter because it typically does not allocate new packets, but just sends out existing ones.
In short you need mempool only for TX. With some luck you even only need them for the skbuff head and a small header buffer, since I would expect iscsi TX to be typically zero copy for data and just passing struct page *s around.
-Andi
Quoted 1 line · Andi Kleen
So in short using mempools on receiving is not needed.
I think you are assuming that there's still some atomic memory available sometime in the future to allocate the skb for the ack, this isn't necessairly true.
I outlined an algo that thanks to proper mempool-like reservation and random picking of all mempool registered on a single nic, will avoid the deadlock for receive. The less mempools there are and the bigger they are, the faster it will recover.
Quoted 1 line · Andi Kleen
So in short using mempools on receiving is not needed.
It is, because you have to ensure that the memory that's needed to receive network packets isn't tied up receiving packets for non-critical sockets, which would leave the critical sockets deadlocked.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 8 lines · Rik van Riel
Quoted 1 line · Andi Kleen
So in short using mempools on receiving is not needed.
It is, because you have to ensure that the memory that's needed to receive network packets isn't tied up receiving packets for non-critical sockets, which would leave the critical sockets deadlocked.
Again the in socket queue is in no way different from all the tens of hundreds of limited size queues that make
up a network. It is quite useless to concentrate on only
one queue in the receiver computer, while all the others still can lose packets.
The only way to solve such problems in the TCP/IP model is to retransmit at the source. This means the TCP write path needs to be reliable, but receiving does not need to be.
TCP will continuing retransmitting for hours. If you network system is so tied up that you cannot receive anything for hours then yes youre screwed, but I doubt memory reservation will fix such extreme problems.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Rik van Riel
Quoted 1 line · Andi Kleen
So in short using mempools on receiving is not needed.
It is, because you have to ensure that the memory that's needed to receive network packets isn't tied up receiving packets for non-critical sockets, which would leave the critical sockets deadlocked.
Quoted 5 lines
Again the in socket queue is in no way different from all the tens of hundreds of limited size queues that make up a network. It is quite useless to concentrate on only one queue in the receiver computer, while all the others still can lose packets.
But ... are the packets already received by the network stack dropped if memory is really low, so we can process the packets for the memor critical sockets ?
If packets received for non-critical sockets can exhaust memory, we will deadlock - and that could be the critical difference between a router (which dumps all packets after some time) and a Linux host running iSCSI...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 16 lines · Andi Kleen
Quoted 8 lines · Rik van Riel
Quoted 1 line · Andi Kleen
So in short using mempools on receiving is not needed.
It is, because you have to ensure that the memory that's needed to receive network packets isn't tied up receiving packets for non-critical sockets, which would leave the critical sockets deadlocked.
Again the in socket queue is in no way different from all the tens of hundreds of limited size queues that make
up a network. It is quite useless to concentrate on only
one queue in the receiver computer, while all the others still can lose packets.
Quoted 3 lines
The only way to solve such problems in the TCP/IP model is to retransmit at the source. This means the TCP write path needs to be reliable, but receiving does not need to be.
You don't seem to understand the deadlock yet. Host is OOM. Host must flush pages to target to free memory. Host manages to draw skbs from private reserve to do its writes. Target acknowledges writes, but host is _still OOM_ and there is no memory including GFP_ATOMIC to allocate receive buffers. Retransmission doesn't help because the acknowledgements will always be dropped.
So it seems we need a way to receive such acknowledgements and quickly discard everything else when we're pushed up against OOM.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 7 lines · Andi Kleen
This does not work because mempools assume you can sleep, and in most NIC drivers you cant while doing RX refill. The NIC drivers can be rewritten to do this refilling in a workqueue. But it is not clear it is useful anyways because Linux failing to allocate a buffer is no different from the network overflowing the hardware queue of the network device, which Linux cannot do anything about.
Actually, not in 2.6 ... we had the same issue in SCSI using mempools for sglist allocation. All of the mempool alocation paths now take gfp_ flags, so you can specify GFP_ATOMIC for interrupt context.
Quoted 9 lines
Basically a network consists of lots of interconnected queues, and even if you try to make the Linux specific side of the queue reliable there are lots of other queues that can still lose packets.
With TCP that is no problem of course because in case of a packet loss the packet is just retransmitted.
So in short using mempools on receiving is not needed.
The object isn't to make the queues *reliable* it's to ensure the system can make forward progress. So all we're trying to ensure is that the sockets used to service storage have some probability of being able to send and receive packets during low memory.
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
James
Quoted 9 lines · James Bottomley
Quoted 7 lines · Andi Kleen
This does not work because mempools assume you can sleep, and in most NIC drivers you cant while doing RX refill. The NIC drivers can be rewritten to do this refilling in a workqueue. But it is not clear it is useful anyways because Linux failing to allocate a buffer is no different from the network overflowing the hardware queue of the network device, which Linux cannot do anything about.
Quoted 3 lines
Actually, not in 2.6 ... we had the same issue in SCSI using mempools for sglist allocation. All of the mempool alocation paths now take
gfp_flags, so you can specifyGFP_ATOMICfor interrupt context.
Just does not work when you are actually short of memory.
Just think a second on how a mempool works: In the extreme case when it cannot allocate system memory anymore it has to wait for someone else to free a memory block into the mempool, then pass it on to the next allocator etc. Basically
it is a direct bypass pipeline for memory to pass memory directly from one high priority user to another. This only works with sleeping. Otherwise you could not handle an arbitary number of users with a single mempool.
So to get a reliable mempool you have to sleep on allocation.
Quoted 4 lines
The object isn't to make the queues *reliable* it's to ensure the system can make forward progress. So all we're trying to ensure is that the sockets used to service storage have some probability of being able to send and receive packets during low memory.
For that it is enough to make the sender reliable. Retransmit takes care of the rest.
Quoted 5 lines
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
Typically because the RX ring of the driver has some packets left.
Also since TCP is very persistent and there is some memory activity left you will have at least occasionally a time slot where a GFP_ATOMIC allocation can succeed.
-Andi
Quoted 15 lines · Andi Kleen
Quoted 3 lines
Actually, not in 2.6 ... we had the same issue in SCSI using mempools for sglist allocation. All of the mempool alocation paths now take
gfp_flags, so you can specifyGFP_ATOMICfor interrupt context.Just does not work when you are actually short of memory.
Just think a second on how a mempool works: In the extreme case when it cannot allocate system memory anymore it has to wait for someone else to free a memory block into the mempool, then pass it on to the next allocator etc. Basically
it is a direct bypass pipeline for memory to pass memory directly from one high priority user to another. This only works with sleeping. Otherwise you could not handle an arbitary number of users with a single mempool.
Quoted 1 line
So to get a reliable mempool you have to sleep on allocation.
But that's not what we use them for. You are confusing reliability with forward progress.
In SCSI we use GFP_ATOMIC mempools in order to make forward progress. All the paths are coded to expect a failure (in which case we requeue). For forward progress what we need is the knowledge that there are n resources out there dedicated to us. When they return they get reallocated straight to us and we can restart the queue processing (there's actually a SCSI trigger that does this).
For receive mempools, the situation is much the same; if you have n reserved buffers, then you have to drop the n+1 th packet. However, the resources will free up and go back to your mempool, and eventually you accept the packet on retransmit.
The killer scenario (and why we require a mempool) is that someone else gets the memory before you but then becomes blocked on another allocation, so now you have no more allocations to allow forward progress.
James
Quoted 7 lines
Quoted 4 lines
The object isn't to make the queues *reliable* it's to ensure the system can make forward progress. So all we're trying to ensure is that the sockets used to service storage have some probability of being able to send and receive packets during low memory.
For that it is enough to make the sender reliable. Retransmit takes care of the rest.
No ... we cannot get down to the situation where GFP_ATOMIC always fails. Now we have no receive capacity at all and the system deadlocks.
Quoted 11 lines
Quoted 5 lines
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
Typically because the RX ring of the driver has some packets left.
Also since TCP is very persistent and there is some memory activity left you will have at least occasionally a time slot where a
GFP_ATOMICallocation can succeed.
That's what I think a mempool is required to guarantee. Without it, there are scenarios where GFP_ATOMIC always fails.
James
Quoted 7 lines · Andi Kleen
Quoted 5 lines
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
Typically because the RX ring of the driver has some packets left.
You can not be sure. Some NICs has very small number for possible HW ring buffers. Under OOM pressure, most likely, host will be so slow that resources might not just be returned back to the HW in time. Though it depends on link-layer driver implementation.
Quoted 7 lines · Dmitry Yusupov
Quoted 6 lines · Andi Kleen
Quoted 5 lines
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
Quoted 7 lines
Quoted 1 line
Typically because the RX ring of the driver has some packets left.
You can not be sure. Some NICs has very small number for possible HW ring buffers. Under OOM pressure, most likely, host will be so slow that resources might not just be returned back to the HW in time. Though it depends on link-layer driver implementation.
I didnt quiet follow the discussion - Let me see if i can phrase the problem correctly (Trying to speak in general terms):
Sender is holding onto memory (retransmit queue i assume) waiting for ACKs. Said sender is under OOM and therefore drops ACKs coming in and as a result cant let go of these precious resource sitting on the retransmit queue.
And iscsi cant wait long enough for someone else to release memory so the ACKs can be delivered.
Did i capture this correctly?
If yes, the solution maybe to just drop all non-high-prio packets coming in during the denial of service attack (for lack of better term). In other words some strict prioritization scheduling (or rate control) at the network level either in the NIC or ingress qdisc level.
On a slightly related topic: is SCSI (not iscsi) considered a reliable protocol?
If yes, why would you wanna run a reliable protocol inside another reliable protocol (TCP)?
cheers,
jamal
Quoted 4 lines · jamal
If yes, the solution maybe to just drop all non-high-prio packets coming in during the denial of service attack (for lack of better term). In other words some strict prioritization scheduling (or rate control) at the network level either in the NIC or ingress qdisc level.
Exactly, that is the proposal. However, we often will need to get the packets off the network card before we can decide whether or not they're high priority.
Also, there can be multiple high priority sockets, and we need to ensure they all make progress. Hence the mempool idea.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 7 lines · Rik van Riel
Quoted 4 lines · jamal
If yes, the solution maybe to just drop all non-high-prio packets coming in during the denial of service attack (for lack of better term). In other words some strict prioritization scheduling (or rate control) at the network level either in the NIC or ingress qdisc level.
Quoted 7 lines
Exactly, that is the proposal. However, we often will need to get the packets off the network card before we can decide whether or not they're high priority.
Also, there can be multiple high priority sockets, and we need to ensure they all make progress. Hence the mempool idea.
I'm sure Rik realizes this, but it's important to note here that "making progress" may require M acknowledgements to N packets representing a single IO. So we need separate send and acknowledge pools for each SO_MEMALLOC socket so that we don't find ourselves wedged with M-1 available mempool slots when we're waiting on ACKs. So accounting ACK packets to the appropriate receiver once we've figured out what socket an ACK is intended for is critical.
Note that ACK here is the application layer command result that needs to be propagated back to the driver (and possibly higher in the case of things like CD writing over iSCSI) and not simply a bit in the TCP header.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 8 lines · Matt Mackall
I'm sure Rik realizes this, but it's important to note here that "making progress" may require M acknowledgements to N packets representing a single IO. So we need separate send and acknowledge pools for each
SO_MEMALLOCsocket so that we don't find ourselves wedged with M-1 available mempool slots when we're waiting on ACKs. So accounting ACK packets to the appropriate receiver once we've figured out what socket an ACK is intended for is critical.
Is this idea discussed or posted somewhere? I just subscribed to the list.
Sounds like what the NICs i described do on rx - some strict priority scheme.
Seems to me the TX side needs to be done early perhaps at the socket layer.
The RX side needs to be done at the NIC or ingress qdisc.
I think there may be need for multiple levels of granularity of priorities for mem allocation pools 8 or more if you want to have different levels of importantance in apps.
The deal with strict prio is the most important apps can eat all the memory if they needed it; so you may need some form of deficit based scheduling or kick in the algorithm only when a certain threshold is crossed system wide.
Quoted 4 lines
Note that ACK here is the application layer command result that needs to be propagated back to the driver (and possibly higher in the case of things like CD writing over iSCSI) and not simply a bit in the TCP header.
I got that (given TCP is stream based).
cheers,
jamal
Quoted 3 lines · Rik van Riel
However, we often will need
to get the packets off the network card before we can decide whether or not they're high priority.
True - although one could argue that with NAPI that decision would be a few opcodes away if you install the ingress qdisc.
So you may end up allocing only to free a few cycles later. Increased memory traffic but the discard happens sufficiently early for a s/ware only solution and CPU cycles not burnt as much.
OTOH, even elcheapo pacific rim nics are beginning to show up with some classifiers in the hardware as well as multiple queues or rx DMA rings. So you could program ACKs or all TCP packets to show up on a higher priority ring and only process that until theres nothing left before processing the low priority ring (i.e dont care if low priority data/app is starved).
Probably someone going out of their way to do high performance iSCSI would consider such hardware.
We dont exactly support multiple rx (or tx) DMA rings however various people seem to be promising patches that work with their hardware (netiron, intel?)
Quoted 3 lines
Also, there can be multiple high priority sockets, and we need to ensure they all make progress. Hence the mempool idea.
Sorry missed the early part of this thread: mempool is some strict priority scheme for mem allocation?
For Sockets: If there was a "control" arbitrator preferably in user space which would install - after a socket open - both network ingress and/or egress rules for prioritization then wouldnt that suffice? The mechanisms are already in place today.
cheers,
jamal
Quoted 11 lines · jamal
Quoted 3 lines · Rik van Riel
However, we often will need
to get the packets off the network card before we can decide whether or not they're high priority.True - although one could argue that with NAPI that decision would be a few opcodes away if you install the ingress qdisc.
So you may end up allocing only to free a few cycles later. Increased memory traffic but the discard happens sufficiently early for a s/ware only solution and CPU cycles not burnt as much.
[...]
I think we first need a software solution that makes no special assumptions about hardware capabilities.
Quoted 6 lines
Quoted 3 lines
Also, there can be multiple high priority sockets, and we need to ensure they all make progress. Hence the mempool idea.
Sorry missed the early part of this thread: mempool is some strict priority scheme for mem allocation?
A mempool is a private allocation pool that attempts to maintain a reserve of N objects. Various users in the kernel already. See mm/mempool.c.
Quoted 3 lines
For Sockets: If there was a "control" arbitrator preferably in user space which would install - after a socket open - both network ingress and/or egress rules for prioritization then wouldnt that suffice?
Generally, we don't want any special handling except when we're effectively OOM.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 4 lines · Matt Mackall
I think we first need a software solution that makes no special assumptions about hardware capabilities.
Absolutely; having hardware assist will typically reduce, not increase, the memory requirements, so the "dumb hardware" solution is likely to be the most generic.
-hpa
Quoted 3 lines
OTOH, even elcheapo pacific rim nics are beginning to show up with some classifiers in the hardware as well as multiple queues or rx DMA rings. So you could program ACKs or all TCP packets to show up on a
Yes, but you end up with limits like "only support upto 4 iscsi connections". which is totally impracticable. The features do not help.
The only feature that might help is an early interrupt where you first process the packet and then later the payload (like the new intel accelerator proposes or some chips have), but that would probably add far too much overhead right now and does not work on most chips anyways.
-Andi
Quoted 9 lines · jamal
I didnt quiet follow the discussion - Let me see if i can phrase the problem correctly (Trying to speak in general terms):
Sender is holding onto memory (retransmit queue i assume) waiting
for ACKs. Said sender is under OOM and therefore drops ACKs coming in
and as a result cant let go of these precious resource sitting on the retransmit queue. And iscsi cant wait long enough for someone else to release memory so the ACKs can be delivered. Did i capture this correctly?
Quoted 4 lines
If yes, the solution maybe to just drop all non-high-prio packets coming in during the denial of service attack (for lack of better term). In other words some strict prioritization scheduling (or rate control) at the network level either in the NIC or ingress qdisc level.
Eventually the TCP will hit its RTX limit and punt the connection, freeing the buffers kept for retransmission right?
Quoted 5 lines
On a slightly related topic: is SCSI (not iscsi) considered a reliable protocol?
If yes, why would you wanna run a reliable protocol inside another reliable protocol (TCP)?
Isn't it better to consider TCP a protocol that provides reliable notice of (presumed) failure rather than a "reliable protocol?"
rick jones
Quoted 4 lines · Rick Jones
Eventually the TCP will hit its RTX limit and punt the connection, freeing the buffers kept for retransmission right?
If i read correctly the people arguing for iscsi say thats not good enough. But they may be having other issues too...
Quoted 9 lines
Quoted 5 lines
On a slightly related topic: is SCSI (not iscsi) considered a reliable protocol?
If yes, why would you wanna run a reliable protocol inside another reliable protocol (TCP)?Isn't it better to consider TCP a protocol that provides reliable notice of (presumed) failure rather than a "reliable protocol?"
You could if the parameters are adequely set (i think).
If both are reliable protocols then they would both have the standard features and parameters:
- transmit (for simplicty assume window of 1)
loop for X times
{
- compute next retransmit time, Y, using some algorithm
- wait for ACK
- timeout
- retransmit
}
so parameters X and retransmit time is where the conflict is. If TCP is eagerly retransmitting a lot of bandwidth could be wasted. if SCSI has X as infinite even more interesting things could happen.
In any case i have seen horror stories of what happened to people who tried to encapsulate an already reliable protocol inside TCP in order to ship it across the big bad internet. I am pretty sure some knowledgeable people were involved in getting iscsi going so it cant be that simple. It would seem preferable to use SCTP instead.
cheers,
jamal
Quoted 8 lines · Jamal
Quoted 5 lines
Eventually the TCP will hit its RTX limit and punt the connection, freeing the buffers kept for retransmission right?
If i read correctly the people arguing for iscsi say thats not good enough. But they may be having other issues too...
It is not good enough for storage.
If we continue to fall back into TCP-will-eventually-recover mentality iSCSI, or at least a "soft" non-offloaded iSCSI over regular non-TOE TCP, will not be able to compete with FC, which uses determinstic credit-based flow control. That non-determinism is a bigger issue, while a corner case like swap device happenning to be iSCSI-remote is just that, a corner case that helps to highlight and bring the general problem to the foreground.
Adding a "critical" or "resource-protected" attribute to the connection context is a step in the right direction. Next steps include:
- triage (closing non-critical connections in OOM);
- socket reopen without deallocating memory (something like: close(int socket_fd, int will_reopen));
- preallocated mempools (it is much better to discover OOM at connection open time than well into runtime).
- better resource "counting" throughout the L3 and L4 layers to preventively handle OOM;
And more incremental steps like that.
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Alex Aizman
If we continue to fall back into TCP-will-eventually-recover mentality iSCSI, or at least a "soft" non-offloaded iSCSI over regular non-TOE TCP, will not be able to compete with FC, which uses determinstic credit-based flow control. That non-determinism is a bigger issue, while a corner case like swap device happenning to be iSCSI-remote is just that, a corner case that helps to highlight and bring the general problem to the foreground.
There is no way to fix the "non-determinism" inherent in a transport. DoS attacks depend on this. The transport has to be deterministic (flow control, QoS) to avoid anything that looks like a DoS (e.g. OOM).
Parallel SCSI suffers the same problem. The priority on the transport is dictated by SCSI ID. Low priority SCSI IDs can (and do) get starved with as few as 5 RAID storage enclosures. The problem is the initiator (host controller) can send data but then like iSCSI, the command times out when the completion doesn't arrive. For parallel SCSI, the SCSI target device never wins SCSI bus arbitration to send back the completion.
People can configure the system so this isn't a problem. But it means not having as much storage per SCSI bus.
hth,
grant
Quoted 5 lines · jamal
In any case i have seen horror stories of what happened to people who tried to encapsulate an already reliable protocol inside TCP in order to ship it across the big bad internet. I am pretty sure some knowledgeable people were involved in getting iscsi going so it cant be that simple. It would seem preferable to use SCTP instead.
Few problems with iSCSI over SCTP I see:
1) It is not as accelerated as TCP yet. But it will eventually. So it is matter of time.
2) There is no working ietf draft for iSCSI over SCTP yet. AFAIK. It will also take few more years.
3) It is not widely used just yet.
Dima
Quoted 7 lines · jamal
Sender is holding onto memory (retransmit queue i assume) waiting for ACKs. Said sender is under OOM and therefore drops ACKs coming in and as a result cant let go of these precious resource sitting on the retransmit queue.
And iscsi cant wait long enough for someone else to release memory so the ACKs can be delivered.
Did i capture this correctly?
Or worse your swap device is on iscsi and you need the ACK to free memory.
But that is unrealistic because it could only happen if 100% of your memory is dirty pages or filled up by other non VM users. Which I think is pretty unlikely. Normally the dirty limits in the VM should prevent it anyways - VM is supposed to block before all your memory is dirty. The CPU can still dirty pages in user space, but the cleaner should also clean it and if necessary block the process.
Quoted 5 lines
If yes, the solution maybe to just drop all non-high-prio packets coming in during the denial of service attack (for lack of better term). In other words some strict prioritization scheduling (or rate control) at the network level either in the NIC or ingress qdisc level.
It does not help. You would need this filtering in all possible queues of the network (including all routers, the RX queue of the NIC etc.) Otherwise the queue in front of you can always starve you in theory.
It is even impossible to do this filtering for normal ethernet devices because they cannot easily distingush different flows and put them into different queus (and even if they can it is usually useless because the max number of flows is so small that you would add an arbitary very small max number limit of your iscsi connections, which users surely would not like)
And you cannot control the Ethernet in front of it neither.
The only exception would be a network that is designed around bandwidth allocation, like an ATM network. But definitely not TCP/IP.
So you cannot solve this problem perfectly. All you can do is to do "good enough" solutions. Have big enough pipes. Keep big enough free memory around etc. I suspect mempools are not really needed for that, probably just some statistical early dropping of packets is enough to give the retransmits a high enough chance to actually make it.
Quoted 4 lines
On a slightly related topic: is SCSI (not iscsi) considered a reliable protocol?
If yes, why would you wanna run a reliable protocol inside another reliable protocol (TCP)?
iSCSI runs on top of TCP AFAIK
-Andi
Quoted 3 lines · Andi Kleen
Which I think is pretty unlikely. Normally the dirty limits in the VM should prevent it anyways - VM is supposed to block before all your memory is dirty. The CPU can still dirty pages in user space,
This is not true for MAP_SHARED as I mentioned earlier in this thread. The dirty limits can't trigger unless you want to take a page fault for every single memory write opcode that touches a clean page (and as well marking the pte clean and wrprotect during writepage). And we're not going to change anything for swap and anon/shm, so it would be still an issue for swap over iscsi.
You can be right the receive path may be less of a pratical issue, but it's still very much an at least theoretical source of deadlock.
Quoted 2 lines · Andrea Arcangeli
You can be right the receive path may be less of a pratical issue, but it's still very much an at least theoretical source of deadlock.
Oh, but this deadlock has been seen in practice. When the system is very low on memory, kswapd will keep allocating memory to write things out, all the way down to 0 free pages.
Then there will be no memory left for GFP_ATOMIC allocations.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 2 lines
You can be right the receive path may be less of a pratical issue, but it's still very much an at least theoretical source of deadlock.
An unsolveable one IMHO. You can just try to be good enough. For that probably simple statistical solutions (like RED on ingres queues and very aggressive freeing of secondary caches like dcache etc.) will be hopefully sufficient.
Basically same thing we do about highmem vs lowmem.
-Andi
Quoted 1 line · Andi Kleen
An unsolveable one IMHO. You can just try to be good enough. For that
I think it's solvable with an algorithm I outlined several emails ago.
Quoted 1 line
Basically same thing we do about highmem vs lowmem.
That's not a deadlock, the lowmem vs highmem issue is about running out of memory too early and getting a -ENOMEM out of a syscall (or a oom-killing).
A few buggy kernels deadlocked in such condition but that just because those buggy kernels would deadlock in most oom condition anyway, current 2.6 kernels and latest 2.4 shouldn't deadlock on a lowmem shortage.
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line · Andi Kleen
An unsolveable one IMHO. You can just try to be good enough. For that
I think it's solvable with an algorithm I outlined several emails ago.
Agreed, this is definately solvable. If people don't agree, we should probably fight this out at the kernel summit. ;)
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 2 lines · Rik van Riel
Agreed, this is definately solvable. If people don't agree, we should probably fight this out at the kernel summit. ;)
So just to make this explicit, we kill this thread and lobby for a topic at the kernel summit discussing the problem of storage over net and the ways of solving it. Yes, I think that's a very good idea.
James
Quoted 10 lines · James Bottomley
Quoted 2 lines · Rik van Riel
Agreed, this is definately solvable. If people don't agree, we should probably fight this out at the kernel summit. ;)
So just to make this explicit, we kill this thread and lobby for a topic at the kernel summit discussing the problem of storage over net and the ways of solving it. Yes, I think that's a very good idea.
That was the whole start of this thread; not just one but two proposed topics that amount to exactly this.
Yes, we need this topic.
-hpa
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line · Andi Kleen
An unsolveable one IMHO. You can just try to be good enough. For that
I think it's solvable with an algorithm I outlined several emails ago.
The problem with you algorithm is that you cannot control how to NIC puts incoming packets into RX rings (and then actually if the packets you are interested in do actually arrive from the net ,-)
While some NICs have hardware support to get high priority packets into different queues these tend to add nasty limits on the max number of connections. Which IMHO is not acceptable.
"We have an enterprise class OS with iSCSI which can only support four swap devices"
-Andi
Quoted 10 lines · Andi Kleen
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line · Andi Kleen
An unsolveable one IMHO. You can just try to be good enough. For that
I think it's solvable with an algorithm I outlined several emails ago.
The problem with you algorithm is that you cannot control how to NIC puts incoming packets into RX rings (and then actually if the packets you are interested in do actually arrive from the net ,-)
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is). If while moving up the stack the skb data doesn't match to the sock->mempool id, we'll just free the packet and put it back in the mempool.
This of course only triggers with skb marked with a mempool ID, all skb allocated with GFP_ATOMIC will have a Null ID and they won't check anything and nothing will change for them.
After GFP fails you pick the skb from a random mempool everytime, so you need all mempools belonging to sockets that routes somehow thorugh a certain NIC driver instance, quickly reacheable from the NIC device driver.
I don't see any problem with this algo. I don't need to control how NIC process the incoming packets, after GFP fails I allocate from a random mempool, I set the skb mempool ID to the ID of the mempool we picked from, and I let the stack process it. Then you need a check as soon as you finished processing the TCP header, to release the skb back in its originating mempool immediatly if the sock mempool ID doesn't match the skb mempool ID but that's easy.
All it matters is that this skb can't get stuck in the middle of nowhere in unfreeable state, but I don't see how could it get stuck in between the netfix_rx and the sock identification via tcp and ip headers. It just can't get stuck, either it's freed prematurely, or it's freed by us with the new mempool id check. It could get stuck if we would let it go ahead into some out of order queues, but not before our new check for mempool ID after tcp header decode.
This is all going to be complex to code, but I think it's technically doable.
Quoted 6 lines
While some NICs have hardware support to get high priority packets into different queues these tend to add nasty limits on the max number of connections. Which IMHO is not acceptable.
"We have an enterprise class OS with iSCSI which can only support four swap devices"
;) I agree the hardware solution isn't appealing.
Quoted 8 lines · Andrea Arcangeli
Quoted 4 lines
The problem with you algorithm is that you cannot control how to NIC puts incoming packets into RX rings (and then actually if the packets you are interested in do actually arrive from the net ,-)
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is).
Mechanisms are in place today.
Quoted 4 lines
If while moving up the stack the skb data doesn't match to the
sock->mempoolid, we'll just free the packet and put it back in the mempool.
I think you may need to reserve some small amount of buffers per NIC (<= RX DMA ring size) that are used as temporary buffers before the decision is made to reassign to the higher priority memory or drop. The decision, if s/ware only, would need to consult a classifier at the ingress (hopefully you are using NAPI and kick this only on overload). The upgrade implies restoring the temporary buffer to the NIC. The NIC rx side only makes progress if temp buffers are available. Since they are restored a short distance after they are allocated (whether you drop or upgrade) progress will always happen
Quoted 3 lines
Quoted 2 lines
"We have an enterprise class OS with iSCSI which can only support four swap devices"
You forgot "carrier-grade" or is that supposed to conflict with "enteprise class"? Cant you be both? ;->
Quoted 1 line
;) I agree the hardware solution isn't appealing.
Well, if a realtek NIC (read: elcheapo, commodity) has such features - in the minimal (regarless of the iscsi problem)- we need to support those features.
cheers,
jamal
Quoted 3 lines · jamal
I think you may need to reserve some small amount of buffers per NIC (<= RX DMA ring size) that are used as temporary buffers before the decision is made to reassign to the higher priority memory or drop.
At that point you also know for which of the higher priority sockets the packet is.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 5 lines
Andrea Arcangeli wrote:
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is).
It makes sense to provide an API for the NIC driver to allocate skb from the *right* mempool. This way if I have plenty of hw rings and/or can allow myself a luxury to associate 1-to-1 connection and ring, there's a nice and clean memory management model. Even NICs that have only few rings could use this - for critical (e.g., storage) connections.
Alex
Quoted 11 lines · Alex Aizman
Quoted 5 lines
Andrea Arcangeli wrote:
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is).
It makes sense to provide an API for the NIC driver to allocate skb from the *right* mempool. This way if I have plenty of hw rings and/or can allow myself a luxury to associate 1-to-1 connection and ring, there's a nice and clean memory management model. Even NICs that have only few rings could use this - for critical (e.g., storage) connections.
It wont work - I can guarantee you that if you add a limit like "we only support 8 iscsi connections max" then users/customers will raise hell because it does not fit their networks.
-Andi
Quoted 3 lines · Andi Kleen
It wont work - I can guarantee you that if you add a limit like "we only support 8 iscsi connections max" then users/customers will raise hell because it does not fit their networks.
What would prevent the iscsi driver from telling the network stack to increase the size of the mempools when additional iscsi connections are configured ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 9 lines · Rik van Riel
Quoted 3 lines · Andi Kleen
It wont work - I can guarantee you that if you add a limit like "we only support 8 iscsi connections max" then users/customers will raise hell because it does not fit their networks.
What would prevent the iscsi driver from telling the network stack to increase the size of the mempools when additional iscsi connections are configured ?
I was talking about the hardware limits for early filtering, not the size of the mempools.
All hardware I found so far has small limits like this.
-Andi
Quoted 2 lines · Andi Kleen
I was talking about the hardware limits for early filtering, not the size of the mempools.
That should not be an issue, since packets in those buffers don't stick around, but can be throw away at will.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 3 lines · Andi Kleen
It wont work - I can guarantee you that if you add a limit like "we only support 8 iscsi connections max" then users/customers will raise hell because it does not fit their networks.
HP has been doing that for years (decades?) for parallel SCSI in "High Availability Configuration Guides". It lays out exactly what is and isn't supported. I'm sure other vendors have similar restrictions. As long as the product is still reasonably useful and the vendor provides a solid assurance it will work, such configuration restrictions are quite acceptable.
I'm NOT arguing "8 iSCSI connections max" is reasonable or enough. I just arguing some sort of limit is acceptable.
grant
Quoted 7 lines
Quoted 4 lines · Andi Kleen
It makes sense to provide an API for the NIC driver to allocate skb from the
*right* mempool. This way if I have plenty of hw rings and/or can allow myself a luxury to associate 1-to-1 connection andring, there's
Quoted 9 lines
Quoted 1 line
a nice and clean memory management model. Even NICs that
have only few
Quoted 1 line
rings could use this - for critical (e.g., storage) connections.
It wont work - I can guarantee you that if you add a limit like "we only support 8 iscsi connections max" then users/customers will raise hell because it does not fit their networks.
Something a bit more intelligent, like: we only support 7 resource-protected (a.k.a. critical) iSCSI connection, and we use one remaining ring for the rest iSCSI, TCP, UDP, etc. traffic. The 7 iSCSI connections could be quite a bit, in terms of LUNs, and just enough for a customer to feel "protected" in a sense that unrelated receive burst starves storage traffic to death. Note that "8 rings" here is just an example; as time goes by the number of hw receive rings and the hw ability to intelligently classify and steer traffic onto these rings will only increase.
Alex
Quoted 8 lines
Something a bit more intelligent, like: we only support 7 resource-protected (a.k.a. critical) iSCSI connection, and we use one remaining ring for the rest iSCSI, TCP, UDP, etc. traffic. The 7 iSCSI connections could be quite a bit, in terms of LUNs, and just enough for a customer to feel "protected" in a sense that unrelated receive burst starves storage traffic to death. Note that "8 rings" here is just an example; as time goes by the number of hw receive rings and the hw ability to intelligently classify and steer traffic onto these rings will only increase.
[assuming you want to solve the oom deadlock 100% which I claim is not practicable. But lets pretend it would be possible with hardware classification support:]
This does not work, because any writable file system can be in theory an OOM deadlock and would need to be resource protected. Just someone needs to mmap a file on it and dirty enough pages in it to full up system memory and make the system deadlock while trying to clean pages to get free memory.
The only "safe" fs that would work is a read only fs where there cant be any dirty pages, but that would be a rather hard to sell restriction.
-Andi
Quoted 5 lines · Andi Kleen
This does not work, because any writable file system can be in theory an OOM deadlock and would need to be resource protected. Just someone needs to mmap a file on it and dirty enough pages in it to full up system memory and make the system deadlock while trying to clean pages to get free memory.
You cannot fill up the mempools with dirty pages, which makes sure we don't deadlock.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 7 lines · Alex Aizman
Quoted 5 lines
Andrea Arcangeli wrote:
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is).
It makes sense to provide an API for the NIC driver to allocate skb from the
That would pretty much need all of the infrastructure for zero copy RX - add all the hooks to update the device driver on local socket hashtable updates. I am sure we will need it at some point and it makes sense, but I suspect it is quite complex work.
-Andi
Quoted 12 lines · Andrea Arcangeli
Quoted 10 lines · Andi Kleen
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line · Andi Kleen
An unsolveable one IMHO. You can just try to be good enough. For that
I think it's solvable with an algorithm I outlined several emails ago.
The problem with you algorithm is that you cannot control how to NIC puts incoming packets into RX rings (and then actually if the packets you are interested in do actually arrive from the net ,-)
Quoted 5 lines
All I care about is to assign a mempool ID to the skb (ID being unique identifier for the tcp connection I don't care how the implementation is). If while moving up the stack the skb data doesn't match to the
sock->mempoolid, we'll just free the packet and put it back in the mempool.
This could still starve on the RX ring level of the hardware which you cant control.
But it might be an improvement, agreed. The problem is that you need lots of infrastructure to tell the driver about TCP connections - it is pretty much near all the work needed for zero copy RX.
Even with all that work it is not the 100% solution some people on this thread seem to be lusting for.
-Andi
Quoted 2 lines · Andi Kleen
This could still starve on the RX ring level of the hardware which you cant control.
It may be inefficient in the recovery, but the point is that it can recover.
Quoted 3 lines
But it might be an improvement, agreed. The problem is that you need lots of infrastructure to tell the driver about TCP connections - it is pretty much near all the work needed for zero copy RX.
The driver only need to have a ring of mempools attached, OK each one is attached to the tcp connection, but the driver won't be required to parse the TCP/IP. After GFP_ATOMIC fails, the driver interurpt handler will pick a skb from a random mempool.
Quoted 2 lines
Even with all that work it is not the 100% solution some people on this thread seem to be lusting for.
I thought it was more than enough, all they care about is not to deadlock anymore, I don't think anybody cares about the performance of the deadlock-scenario.
I agree with Jamal that his suggestion to use an high-per ring is very good (I didn't even know some card supported this feature), so if somebody wants the deadlock scenario not to run in "degraded mode", they will have to use some more advanced hardware the way Jamal is suggesting (or get rid of TCP all together and use TCP/IP offload with the security risks it introduces or RDMA or whatever other point to point high perf DMA technology like quadrix etc..).
I suspect the deadlock scenario is infrequent enough that it won't matter how fast it recovers as long as it eventually does.
Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines
Even with all that work it is not the 100% solution some people on this thread seem to be lusting for.
I thought it was more than enough, all they care about is not to deadlock anymore, I don't think anybody cares about the performance of the deadlock-scenario.
True. this is all we need to make "soft" iSCSI viable alternative to FC. Btw, other OSes can do that today.
Quoted 7 lines
I agree with Jamal that his suggestion to use an high-per ring is very good (I didn't even know some card supported this feature), so if somebody wants the deadlock scenario not to run in "degraded mode", they will have to use some more advanced hardware the way Jamal is suggesting (or get rid of TCP all together and use TCP/IP offload with the security risks it introduces or RDMA or whatever other point to point high perf DMA technology like quadrix etc..).
One good example is Neterion 10Gbps card. It supports up to 8 priority rings. Could someone point me to the API which driver could utilize to configure ring's priorities with 2.6.x? Thanks.
Quoted 12 lines · Andi Kleen
Quoted 7 lines · jamal
Sender is holding onto memory (retransmit queue i assume) waiting for ACKs. Said sender is under OOM and therefore drops ACKs coming in and as a result cant let go of these precious resource sitting on the retransmit queue.
And iscsi cant wait long enough for someone else to release memory so the ACKs can be delivered.
Did i capture this correctly?Or worse your swap device is on iscsi and you need the ACK to free memory.
Quoted 7 lines
But that is unrealistic because it could only happen if 100% of your memory is dirty pages or filled up by other non VM users. Which I think is pretty unlikely. Normally the dirty limits in the VM should prevent it anyways - VM is supposed to block before all your memory is dirty. The CPU can still dirty pages in user space, but the cleaner should also clean it and if necessary block the process.
I seem to recall this being fairly easy to trigger by simply pulling the network cable while there's heavy mmap + write load. The system will quickly spiral down into OOM and will remain wedged when you plug the network back in. With iSCSI, after some extended period all the I/Os will have SCSI timeouts and lose everything.
It's going to be fairly typical for iSCSI boxes to do all their I/O over iSCSI, including swap and root. Things like blades and cluster nodes.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 5 lines · Matt Mackall
I seem to recall this being fairly easy to trigger by simply pulling the network cable while there's heavy mmap + write load. The system will quickly spiral down into OOM and will remain wedged when you plug the network back in. With iSCSI, after some extended period all the I/Os will have SCSI timeouts and lose everything.
We've discussed that already. SCSI timeout logic just doesn't fit. (see rfc3720). For iSCSI, SCSI timeout logic *must* be disabled until iSCSI recovery is complete. host block/unblock logic in recent iSCSI transport patch will help to implement that.
Quoted 9 lines · Dmitry Yusupov
Quoted 5 lines · Matt Mackall
I seem to recall this being fairly easy to trigger by simply pulling the network cable while there's heavy mmap + write load. The system will quickly spiral down into OOM and will remain wedged when you plug the network back in. With iSCSI, after some extended period all the I/Os will have SCSI timeouts and lose everything.
Quoted 3 lines
We've discussed that already. SCSI timeout logic just doesn't fit. (see rfc3720). For iSCSI, SCSI timeout logic *must* be disabled until iSCSI recovery is complete.
This has actually been brought up when the scsi_times_out thread was going on. It kinda was at least. Some driver writers inluding sfnet used to play a lot of tricks with the timers to accomplish this (this was the goal for sfnet at least) and I do not think linux-scsi will allow it. Maybe that will change.
host block/unblock logic in recent iSCSI transport
Quoted 2 lines
patch will help to implement that.
No it won't :( block/unblock does not disable timeouts it just makes it so new commands are not queued and they timeout when the driver knows that the transport is hosed.
Quoted 2 lines · jamal
On a slightly related topic: is SCSI (not iscsi) considered a reliable protocol?
Yes and No. "SCSI" covers several layers of the ISO networking model. Parallel SCSI transport is reliable.
Quoted 2 lines
If yes, why would you wanna run a reliable protocol inside another reliable protocol (TCP)?
Think of SCSI command protocol more like NFS: Just a way to send commands/data to a "storage device".
grant
Quoted 8 lines · Dmitry Yusupov
Quoted 6 lines · Andi Kleen
Quoted 5 lines
In your scenario, if we're out of memory and the system needs several ACK's to the swap device for pages to be released to the system, I don't see how we make forward progress since without a reserved resource to allocate from how does the ack make it up the stack to the storage driver layer?
Quoted 8 lines
Quoted 1 line
Typically because the RX ring of the driver has some packets left.
You can not be sure. Some NICs has very small number for possible HW ring buffers. Under OOM pressure, most likely, host will be so slow that resources might not just be returned back to the HW in time. Though it depends on link-layer driver implementation.
This seems to become part of the whole thing... in other words, in an OOM situation, may have to free RX ring entries by just dropping packets as necessary...
-hpa
Quoted 35 lines · Andi Kleen
Quoted 5 lines · Rik van Riel
one for sending and one for receiving
2) have a global emergency mempool available to receive network packets whenGFP_ATOMICfails - this is useful since we don't know who a packet is for when we get the NIC interrupt, and it's easy to have just one pool to checkThis does not work because mempools assume you can sleep, and in most NIC drivers you cant while doing RX refill. The NIC drivers can be rewritten to do this refilling in a workqueue. But it is not clear it is useful anyways because Linux failing to allocate a buffer is no different from the network overflowing the hardware queue of the network device, which Linux cannot do anything about.
Basically a network consists of lots of interconnected queues, and even if you try to make the Linux specific side of the queue reliable there are lots of other queues that can still lose packets.
With TCP that is no problem of course because in case of a packet loss the packet is just retransmitted.
So in short using mempools on receiving is not needed.
Now memory allocation for writing is a different chapter. You cannot recover from a lost writing.
The kernel currently goes even in endless loops in this case (e.g. on TCP FIN allocation)With the exception of routing the allocation fortunately usually happens there in:
- socket context: great you can use a per socket mempool
- thread context: you can sleep
Quoted 8 lines
With routing that is not the case, but it does not matter because it typically does not allocate new packets, but just sends out existing ones.
In short you need mempool only for TX. With some luck you even only need them for the skbuff head and a small header buffer, since I would expect iscsi TX to be typically zero copy for data and just passing struct page *s around.
that's always true for Open-iSCSI and for sfnet in FFP(rcf3720) but not for other implementations. moreover, if NIC does not support checksumming on Tx than sendpage() interface will fail to sendmsg() which will allocate new skb each time.
Let me slightly hijack this thread to throw out another topic that I think is worth talking about at the kernel summit: handling remote DMA (RDMA) network technologies.
As some of you might know, I'm one of the main authors of the InfiniBand support in the kernel, and I think we have things fairly well in hand there, although handling direct userspace access to RDMA capabilities may raise some issues worth talking about.
However, there is also RDMA-over-TCP hardware beginning to be used, based on the specs from the IETF rddp working group and the RDMA Consortium. I would hope that we can abstract out the common pieces for InfiniBand and RDMA NIC (RNIC) support and morph drivers/infiniband into a more general drivers/rdma.
This is not _that_ offtopic, since RDMA NICs provide another way of handling OOM for iSCSI. By having the NIC handle the network transport through something like iSER, you avoid a lot of the issues in this thread. Having to reconnect to a target while OOM is still a problem, but it seems no worse in principal than the issues with a dump FC card that needs the host driver to handling fabric login.
I know that in the InfiniBand world, people have been able to run stress tests of storage over SCSI RDMA Protocol (SRP) with very heavy swapping going on and no deadlocks. SRP is in effect network storage with the transport handled by the IB hardware.
However there are some sticky points that I would be interested in discussing. For example, the IETF rddp drafts envisage what they call a "dual stack" model: TCP connections are set up by the usual network stack and run for a while in "streaming" mode until the application is ready to start using RDMA. At that point there is an "MPA" negotiation and then the socket is handed over to the RNIC. Clearly moving the state from the kernel's stack to the RNIC is not trivial.
Other developers who have more direct experience with RNIC hardware or perhaps just strong opinions may have other things in this area that they'd like to talk about.
Thanks,
Roland
Re: Topic: Remote DMA network technologies
[ Can we start updating the Subject line occasionally when we have
a specific topic, like above. And no, I'm not top-posting, I'm commenting on the Subject:. ;-) --gerrit ]
Quoted 44 lines · Roland Dreier
Let me slightly hijack this thread to throw out another topic that I think is worth talking about at the kernel summit: handling remote DMA (RDMA) network technologies.
As some of you might know, I'm one of the main authors of the InfiniBand support in the kernel, and I think we have things fairly well in hand there, although handling direct userspace access to RDMA capabilities may raise some issues worth talking about.
However, there is also RDMA-over-TCP hardware beginning to be used, based on the specs from the IETF rddp working group and the RDMA Consortium. I would hope that we can abstract out the common pieces for InfiniBand and RDMA NIC (RNIC) support and morph
drivers/infinibandinto a more generaldrivers/rdma.This is not _that_ offtopic, since RDMA NICs provide another way of handling OOM for iSCSI. By having the NIC handle the network transport through something like iSER, you avoid a lot of the issues in this thread. Having to reconnect to a target while OOM is still a problem, but it seems no worse in principal than the issues with a dump FC card that needs the host driver to handling fabric login.
I know that in the InfiniBand world, people have been able to run stress tests of storage over SCSI RDMA Protocol (SRP) with very heavy swapping going on and no deadlocks. SRP is in effect network storage with the transport handled by the IB hardware.
However there are some sticky points that I would be interested in discussing. For example, the IETF rddp drafts envisage what they call a "dual stack" model: TCP connections are set up by the usual network stack and run for a while in "streaming" mode until the application is ready to start using RDMA. At that point there is an "MPA" negotiation and then the socket is handed over to the RNIC. Clearly moving the state from the kernel's stack to the RNIC is not trivial.
Other developers who have more direct experience with RNIC hardware or perhaps just strong opinions may have other things in this area that they'd like to talk about.
Thanks,
Roland
Re: Topic: Remote DMA network technologies
Gerrit> [ Can we start updating the Subject line occasionally when Gerrit> we have a specific topic, like above. And no, I'm not Gerrit> top-posting, I'm commenting on the Subject:. ;-) --gerrit ]
Sorry -- your point is well taken, but everyone else was having so much fun discussing iSCSI specifics under the old Subject line that I just went with the flow ;)
- Roland
A message that is not in the archive (sent off-list or not yet received): 1112042936.5088.22.camel@beastie
Quoted 3 lines · Dmitry Yusupov
If you have plans to start new project such as SoftRDMA than yes. lets discuss it since set of problems will be similar to what we've got with software iSCSI Initiators.
I'm somewhat interested in seeing a SoftRDMA project get off the ground. At least the NatSemi 83820 gige MAC is able to provide early-rx interrupts that allow one to get an rx interrupt before the full payload has arrived making it possible to write out a new rx descriptor to place the payload wherever it is ultimately desired. It would be fun to work on if not the most performant RDMA implementation.
Quoted 4 lines
I'm not a believer in any HW state-full protocol offloading technologies and that was one of my motivations to initiate Open-iSCSI project to prove that performance is not an issue anymore. And we succeeded, by showing comparable to iSCSI HW Initiator's numbers.
Agreed. After working on a full TOE implementation, I think that the niche market most TOE vendors are pursuing is not one that the Linux community will ever develop for. Hardware vendors that gradually add offloading features from the NIC realm to speed up the existing network stack are a much better fit with Linux.
Quoted 4 lines
Though, for me, RDMA over TCP is an interesting topic from software implementation point of view. I was thinking about organizing new project. If someone knows that related work is already started - let me know since I might be interested to help.
Shall we create a new mailing list? I guess it's time to update majordomo... =)
-ben
Linux support for RDMA (was: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit Proposed Topics)
Benjamin> Agreed. After working on a full TOE implementation, I Benjamin> think that the niche market most TOE vendors are Benjamin> pursuing is not one that the Linux community will ever Benjamin> develop for. Hardware vendors that gradually add Benjamin> offloading features from the NIC realm to speed up the Benjamin> existing network stack are a much better fit with Linux.
I have to admit I don't know much about the TOE / RDMA/TCP / RNIC (or whatever you want to call it) world. However I know that the large majority of InfiniBand use right now is running on Linux, and I hope the Linux community is willing to work with the IB community.
InfiniBand adoption is strong right now, with lots of large clusters being built. It seems reasonable that RDMA/TCP should be able to compete in the same market. Whether InfiniBand or RDMA/TCP or both will survive or prosper is a good question, and I think it's too early to tell yet.
- R.
Re: Linux support for RDMA (was: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit Proposed Topics)
Quoted 11 lines · Roland Dreier
Benjamin> Agreed. After working on a full TOE implementation, I Benjamin> think that the niche market most TOE vendors are Benjamin> pursuing is not one that the Linux community will ever Benjamin> develop for. Hardware vendors that gradually add Benjamin> offloading features from the NIC realm to speed up the Benjamin> existing network stack are a much better fit with Linux.
I have to admit I don't know much about the TOE / RDMA/TCP / RNIC (or whatever you want to call it) world. However I know that the large majority of InfiniBand use right now is running on Linux, and I hope the Linux community is willing to work with the IB community.
My comments were more directed to Full TOE implementations, which tend to suffer from incomplete feature coverage if compared to the native Linux TCP/IP stack. Wedging a complete network stack onto a piece of hardware does allow for better performance characteristics on workloads where the networking overhead matters, but it comes at the cost of not being able to trivially change the resulting stack. Plus there are very few vendors who are willing to release firmware code to the open source community.
Quoted 5 lines
InfiniBand adoption is strong right now, with lots of large clusters being built. It seems reasonable that RDMA/TCP should be able to compete in the same market. Whether InfiniBand or RDMA/TCP or both will survive or prosper is a good question, and I think it's too early to tell yet.
I'm curious how the 10Gig ethernet market will pan out. Time and again the market has shown that ethernet always has the cost advantage in the end. If something like Intel's I/O Acceleration Technology makes it that much easier for commodity ethernet to achieve similar performance characteristics over ethernet to that of IB and fibre channel, the cost advantage alone might switch some new customers over. But the hardware isn't near what IB offers today, making IB an important niche filler.
-ben
Signature · 1 line
"Time is what keeps everything from happening all at once." -- John Wheeler
Re: Linux support for RDMA
Quoted 3 lines · Benjamin LaHaise
I'm curious how the 10Gig ethernet market will pan out. Time and again the market has shown that ethernet always has the cost advantage in the end. If something like Intel's I/O Acceleration Technology makes it that much easier for commodity ethernet to achieve similar performance characteristics over ethernet to that of IB and fibre channel, the cost advantage alone might switch some new customers over. But the hardware isn't near what IB offers today, making IB an important niche filler.
From what I've seen coming down the pipe, I think 10GE is going to eventually win over IB, just like previous generations did over Token Ring, FDDI and other niche filler technologies. It doesn't, as you say, mean that e.g. IB doesn't matter *now*; furthermore, it also matters for the purpose of fixing the kind of issues that are going to have to be fixed anyway.
-hpa
Quoted 11 lines · Benjamin LaHaise
Quoted 3 lines · Dmitry Yusupov
If you have plans to start new project such as SoftRDMA than yes. lets discuss it since set of problems will be similar to what we've got with software iSCSI Initiators.
I'm somewhat interested in seeing a SoftRDMA project get off the ground. At least the NatSemi 83820 gige MAC is able to provide early-rx interrupts that allow one to get an rx interrupt before the full payload has arrived making it possible to write out a new rx descriptor to place the payload wherever it is ultimately desired. It would be fun to work on if not the most performant RDMA implementation.
I see a lot of skepticism around early-rx interrupt schema. It might work for gige, but i'm not sure if it will fit into 10g.
What RDMA gives us is zero-copy on receive and new networking api which has a potential to be HW accelerated. SoftRDMA will never avoid copying on receive. But benefit for SoftRDMA would be its availability on client sides. It is free and it could be easily deployed. Soon Intel & Co will give us 2,4,8... multi-core CPUs for around 200$ :), So, who cares if one of those cores will do receive side copying?
Quoted 16 lines · Dmitry Yusupov
Quoted 11 lines · Benjamin LaHaise
Quoted 3 lines · Dmitry Yusupov
If you have plans to start new project such as SoftRDMA than yes. lets discuss it since set of problems will be similar to what we've got with software iSCSI Initiators.
I'm somewhat interested in seeing a SoftRDMA project get off the ground. At least the NatSemi 83820 gige MAC is able to provide early-rx interrupts that allow one to get an rx interrupt before the full payload has arrived making it possible to write out a new rx descriptor to place the payload wherever it is ultimately desired. It would be fun to work on if not the most performant RDMA implementation.
I see a lot of skepticism around early-rx interrupt schema. It might work for gige, but i'm not sure if it will fit into 10g.
Quoted 7 lines
What RDMA gives us is zero-copy on receive and new networking api which has a potential to be HW accelerated. SoftRDMA will never avoid copying on receive. But benefit for SoftRDMA would be its availability on client sides. It is free and it could be easily deployed. Soon Intel & Co will give us 2,4,8... multi-core CPUs for around 200$ :), So, who cares if one of those cores will do receive side copying?
dedicated core to dealing with interrupt is fine. but the memory bandwidth is still over-used right?
ming
Quoted 1 line · Dmitry Yusupov
So, who cares if one of those cores will do receive side copying?
It burns backplane bandwidth that could be used for other things. The problem isn't the CPU cycles. It's the number
of times the data has to cross the memory bus.
grant
Quoted 6 lines
What RDMA gives us is zero-copy on receive and new networking api which has a potential to be HW accelerated. SoftRDMA will never avoid copying on receive. But benefit for SoftRDMA would be its availability on client sides. It is free and it could be easily deployed. Soon Intel & Co will give us 2,4,8... multi-core CPUs for around 200$ :), So, who cares if one of those cores will do receive side copying?
20 years ago, in certain circles at least, people were saying "With 32-bits of addressing, who cares if we allocate much memory" :)
Speaking a bit more prosaicly, if that core is sitting there churning through data copies, what affect does that have on the rest of the bus(ses) and the memory? What else will the client want to be able to push around that those data copies may preclude?
rick jones
Linux support for RDMA (was: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit Proposed Topics)
In reply to a message not in the archiveDmitry> Basically, HW offloading all kind of is a different Dmitry> subject. Yes, iSER/RDMA/RNIC will help to avoid bunch of Dmitry> problems but at the same time will add bunch of new Dmitry> problems. OOM/deadlock problem we are discussing is a Dmitry> software, *not* hardware related.
Yes, that's why I said I was hijacking the topic to bring up something else I was interested in :)
Dmitry> If you have plans to start new project such as SoftRDMA Dmitry> than yes. lets discuss it since set of problems will be Dmitry> similar to what we've got with software iSCSI Initiators.
No, I don't have plans for such a project, although I would be interesting in participating in a small way. Unfortunately I'm involved in too many other things on to do much real work.
My main interest comes from the InfiniBand world. Right now we have the beginnings of good support for IB in drivers/infiniband, but people are always talking to me about adding support for RDMA/TCP hardware. I think we should be able to evolve the curent InfiniBand API to a more generic RDMA API, and I would hope that a "SoftRDMA" project can fit in as just another low-level device driver (soft of the same way software iSCSI sits under the SCSI stack).
In fact I think SoftRDMA would be very good for this generalization work, as it would force us to come up with very flexible APIs.
Dmitry> I'm not a believer in any HW state-full protocol Dmitry> offloading technologies and that was one of my motivations Dmitry> to initiate Open-iSCSI project to prove that performance Dmitry> is not an issue anymore. And we succeeded, by showing Dmitry> comparable to iSCSI HW Initiator's numbers.
Fair enough. I think I agree that HW offload is not really justified if all you care about is storage, although a cheap iSCSI HBA than handles all the transport and just lets the host queue IOs seems like a reasonable thing to put in a server that has work to do beyond running a storage stack.
It seems that many people are using RDMA hardware (mostly InfiniBand now, maybe RDMA/TCP will catch on) hardware for other reasons. In those cases users often want to share the same fabric and NIC for storage too. But my main interest right now is in getting RDMA working well on Linux for the users that are already out there -- I know many IB clusters with hundreds and even thousands of nodes are being built all the time, so InfiniBand must be solving some real problems for users.
- R.