Skip to contentSkip to conversation

ipvs network name space (netns) aware

RFC PATCH In review Hans Schillstrom 3 participants 5 messages Oct 8, 2010 – Oct 20, 2010

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover ipvs network name space (netns) aware

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns. For users that don't use network name space (netns) this patch is completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools) i.e. a light weight virtualization. For example it's possible to run one or several lvs on a real server in their own network name spaces. From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
- Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h. What is moved and what is not ?

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?
Should more lock variables be moved (or less) ?

PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010 and [patch v4] ipvs: IPv6 tunnel mode

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

SUMMARY

 include/net/ip_vs.h                     |  136 ++++---
 include/net/net_namespace.h             |    2 +
 include/net/netns/ip_vs.h               |  112 +++++
 net/netfilter/ipvs/ip_vs_app.c          |   96 +++--
 net/netfilter/ipvs/ip_vs_conn.c         |  296 ++++++++-----
 net/netfilter/ipvs/ip_vs_core.c         |  155 ++++---
 net/netfilter/ipvs/ip_vs_ctl.c          |  771 +++++++++++++++++--------------
 net/netfilter/ipvs/ip_vs_est.c          |  127 +++--
 net/netfilter/ipvs/ip_vs_ftp.c          |   64 ++-
 net/netfilter/ipvs/ip_vs_proto.c        |  108 +++++-
 net/netfilter/ipvs/ip_vs_proto_ah_esp.c |   34 +-
 net/netfilter/ipvs/ip_vs_proto_sctp.c   |  126 +++---
 net/netfilter/ipvs/ip_vs_proto_tcp.c    |  112 +++--
 net/netfilter/ipvs/ip_vs_proto_udp.c    |  107 +++--
 net/netfilter/ipvs/ip_vs_sync.c         |  323 +++++++------
 net/netfilter/ipvs/ip_vs_xmit.c         |   12 +-
 16 files changed, 1588 insertions(+), 993 deletions(-)

The patch will be divided for readability into
1. include files
2. app
3. conn
4. core
5. ctl
6. est
7. ftp
8. proto files
9. sync and xmit

Include files,
A new file added include/net/netns/ip_vs.h containg all netns specific data. include/net/net_namespce.h, pointer to "struct ipvs" added. include/net/ip_vs.h a new struct added, and many prototypes changed.

* ip_vs_core.c
All netns init origins from this file - ip_vs_init()

* ip_vs_conn.c
Lock array for conn table is kept due to performance, (or am I wrong here ?).
"static struct ip_vs_aligned_lock __ip_vs_conntbl_lock_array[CT_LOCKARRAY_SIZE] __cacheline_aligned;"

* ip_vs_ctl.c
drop_ rate is still global

TESTING
This patch have been running for a month now with three LVS/machine one in root name-space and two in other name-space.
Both IPv4 & IPv6 have been tested in all three modes DR/TUN and NAT Only a limited set of algos have been used (read rr).

Backup have been there all the time and a switch has been performed a couple of times.

There is still some BUG_ON(!net) left ...

Not tested yet:
Drop level, DOS, schedulers, performance ....
Netns exit after usage of LVS (due to a bug in netdev/ipip somewhere tunl0 and

--
Regards
Hans Schillstrom <[email protected]>

maintainer
#1 lore raw
Quoted 24 lines · Hans Schillstrom

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns. For users that don't use network name space (netns) this patch is completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools) i.e. a light weight virtualization. For example it's possible to run one or several lvs on a real server in their own network name spaces.

Quoted 1 line

From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
- Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h. What is moved and what is not ?

Quoted 6 lines

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?

My gut-feeling is that per netns makes more sense.

Quoted 1 line

Should more lock variables be moved (or less) ?

I'm unsure what you are asking here but I will make a general statement about locking in IPVS: it needs work.

Quoted 6 lines

PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010 and [patch v4] ipvs: IPv6 tunnel mode

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

Unfortunately the patches don't apply with the persistence engine patches which were recently merged into nf-next-2.6 (although "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).

I'm happy to work with you to make the required changes there.

(I realise those patches weren't merged when you made your post.
But regardless, either your or me will need to update the patches).

Another issue is that your patches seem to be split in a way where the build breaks along the way. E.g. after applying patch 1, the build breaks. Could you please split things up in a manner such that this doesn't happen. The reason being that it breaks bisection.

Lastly, could you provide a unique subject for each patch. I know its a bit tedious, but it does make a difference when browsing the changelog.

#2 lore raw
Quoted 25 lines · On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
Quoted 24 lines · Hans Schillstrom

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns. For users that don't use network name space (netns) this patch is completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools) i.e. a light weight virtualization. For example it's possible to run one or several lvs on a real server in their own network name spaces.

Quoted 1 line

From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
- Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h. What is moved and what is not ?

Quoted 6 lines
Quoted 6 lines

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?

This is a tricky one (I think),
if the interface is shared with root name-space and/or other name-spaces
- use grand total
if it's an "own interface"
- drop rate can/should be in netns...

Quoted 7 lines

My gut-feeling is that per netns makes more sense.

Quoted 1 line

Should more lock variables be moved (or less) ?

I'm unsure what you are asking here but I will make a general statement about locking in IPVS: it needs work.

Some locks still resides as global variables, and others in netns_ipvs struct. Since you have a lot of experience with IPVS locks,
you might have ideas what to move and what to not move.

Quoted 6 lines
Quoted 6 lines

PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010 and [patch v4] ipvs: IPv6 tunnel mode

Quoted 6 lines
Quoted 1 line

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

Unfortunately the patches don't apply with the persistence engine patches which were recently merged into nf-next-2.6 (although "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).

I do have a patch based on the nf-next without the SIP/PE patch

Quoted 1 line

I'm happy to work with you to make the required changes there.

I would appreciate that.

Quoted 10 lines

(I realise those patches weren't merged when you made your post.
But regardless, either your or me will need to update the patches).

Another issue is that your patches seem to be split in a way where the build breaks along the way. E.g. after applying patch 1, the build breaks. Could you please split things up in a manner such that this doesn't happen. The reason being that it breaks bisection.

Hmm, Daniel also pointed at this,
The Patch is quite large, and will become even larger with pe and sip.
My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it. I don't know that might be a stupid ?
It's hard to break it up, making the code reentrant causes changes every where.

Daniel L, had another approach break it into many many tiny patches.

Quoted 4 lines

Lastly, could you provide a unique subject for each patch. I know its a bit tedious, but it does make a difference when browsing the changelog.

Yepp, no problem

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Regards
Hans Schillstrom <[email protected]>
#3 lore raw
Quoted 85 lines · Hans Schillstrom
Quoted 26 lines · On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
Quoted 24 lines · Hans Schillstrom

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns. For users that don't use network name space (netns) this patch is completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools) i.e. a light weight virtualization. For example it's possible to run one or several lvs on a real server in their own network name spaces.

Quoted 1 line

From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
- Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h. What is moved and what is not ?

Quoted 7 lines
Quoted 7 lines

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?

This is a tricky one (I think),
if the interface is shared with root name-space and/or other name-spaces
- use grand total
if it's an "own interface"
- drop rate can/should be in netns...

Quoted 8 lines

My gut-feeling is that per netns makes more sense.

Quoted 2 lines

Should more lock variables be moved (or less) ?

I'm unsure what you are asking here but I will make a general statement about locking in IPVS: it needs work.

Some locks still resides as global variables, and others in netns_ipvs struct. Since you have a lot of experience with IPVS locks,
you might have ideas what to move and what to not move.

Quoted 5 lines
Quoted 5 lines

PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010 and [patch v4] ipvs: IPv6 tunnel mode

Quoted 7 lines
Quoted 2 lines

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

Unfortunately the patches don't apply with the persistence engine patches which were recently merged into nf-next-2.6 (although "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).

I do have a patch based on the nf-next without the SIP/PE patch

Quoted 2 lines

I'm happy to work with you to make the required changes there.

I would appreciate that.

Quoted 10 lines

(I realise those patches weren't merged when you made your post.
But regardless, either your or me will need to update the patches).

Another issue is that your patches seem to be split in a way where the build breaks along the way. E.g. after applying patch 1, the build breaks. Could you please split things up in a manner such that this doesn't happen. The reason being that it breaks bisection.

Quoted 8 lines

Hmm, Daniel also pointed at this,
The Patch is quite large, and will become even larger with pe and sip.
My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it. I don't know that might be a stupid ?
It's hard to break it up, making the code reentrant causes changes every where.

Daniel L, had another approach break it into many many tiny patches.

These patches can be folded if the result is an acceptable patch size. The more important is to not break the init_net. Introducing the namespace concept step by step, from the bottom-up in the call stack is good because we focus first on making the namespace available in the code while keeping the init_net functional. The small patches help to git bisect and spot the patch which lead to a regression. As the patches are small, it is easy to quickly find the culprit and fix it, especially for the people who are not author of these patches.

During this step of namespace introduction, we prevent to unshare the network for ipvs, with a non-op when the namespace is not init_net.

When everything is ready and the init_net is running without problem, we remove the non-op and consolidate the namespace part of ipvs.

This is how I am used to do, but that does not imply it is the best way to do that :)

Quoted 15 lines
Quoted 5 lines

Lastly, could you provide a unique subject for each patch. I know its a bit tedious, but it does make a difference when browsing the changelog.

Yepp, no problem

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#4 lore raw
In reply to Hans Schillstrom
Quoted 30 lines · Hans Schillstrom
Quoted 29 lines · On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
Quoted 28 lines · Hans Schillstrom

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns. For users that don't use network name space (netns) this patch is completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools) i.e. a light weight virtualization. For example it's possible to run one or several lvs on a real server in their own network name spaces.

Quoted 1 line

From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
- Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h. What is moved and what is not ?

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?

This is a tricky one (I think),
if the interface is shared with root name-space and/or other name-spaces
- use grand total
if it's an "own interface"
- drop rate can/should be in netns...

I hadn't thought about shared devices - yes that is tricky.

Quoted 10 lines
Quoted 6 lines

My gut-feeling is that per netns makes more sense.

Quoted 1 line

Should more lock variables be moved (or less) ?

I'm unsure what you are asking here but I will make a general statement about locking in IPVS: it needs work.

Some locks still resides as global variables, and others in netns_ipvs struct. Since you have a lot of experience with IPVS locks,
you might have ideas what to move and what to not move.

My basic thought is that locks tend to either related to a connection or the configuration of a service. And it seems to me that if you have a per-namespace connection hash table then both of these categories of locks are good candidates to be made per-namespace.

Do you have any particular locks that you are worried about?

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010 and [patch v4] ipvs: IPv6 tunnel mode

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

Unfortunately the patches don't apply with the persistence engine patches which were recently merged into nf-next-2.6 (although "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).

Quoted 5 lines

I do have a patch based on the nf-next without the SIP/PE patch

Quoted 1 line

I'm happy to work with you to make the required changes there.

I would appreciate that.

No problem. I am a bit busy this week as I am attending the Netfilter Workshop. But I will try to find some time to rebase your changes soon.

Quoted 9 lines
Quoted 9 lines

(I realise those patches weren't merged when you made your post.
But regardless, either your or me will need to update the patches).

Another issue is that your patches seem to be split in a way where the build breaks along the way. E.g. after applying patch 1, the build breaks. Could you please split things up in a manner such that this doesn't happen. The reason being that it breaks bisection.

Quoted 7 lines

Hmm, Daniel also pointed at this,
The Patch is quite large, and will become even larger with pe and sip.
My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it. I don't know that might be a stupid ?
It's hard to break it up, making the code reentrant causes changes every where.

Daniel L, had another approach break it into many many tiny patches.

I would prefer the tiny patch approach.

Quoted 5 lines
Quoted 4 lines

Lastly, could you provide a unique subject for each patch. I know its a bit tedious, but it does make a difference when browsing the changelog.

Yepp, no problem