Skip to contentSkip to conversation

IPVS: Backup Adding Ipv6 and Persistence support

PATCH In review Hans Schillstrom 3 participants 4 messages Nov 12, 2010 – Nov 15, 2010 IPVS

lore
Share & tools
lore
#0 lore raw

v3 cover IPVS: Backup Adding Ipv6 and Persistence support

This patch series adds/(updates) the following functionality in the synchronization between master and backup daemons.

- IPv6
- Persistence Engine
- Firewall marks transferred
- Time-outs transferred.
- Flag field increased to 32 bits.

Note:
This patches depends on Patch 1-9 in staging branch:
"IPVS: Backup, Prepare for transferring firewall marks (fwmark) to the backup daemon." to ...
"IPVS: skb defrag in L7 helpers"

A new message format is also introduced, not understood by old backup daemons. For compatibility reasons receiving the old version (version 0) is still possible. Old (version 0) backups will just drop new (Version 1) messages.
It's also possible to send sync msg in version 0 format, by sysctl #sysclt -w net.ipv4.vs.sync_version=0

Update scenario:
Update the Machine with Bakup daemon first.

Message structure:
A new 32 bit word is added to the header,
where the old count_Conns is set to 0 so old Backup daemons just will drop the packet. Added is version (1) and there is also a spare 16 bit field at the end. This is a reworked version based upon Simon Hormans work.

Version 1:
Note, first byte should be Zero, so ver 0 receivers will drop the packet.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0 | SyncID | Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Count Conns | Version | Reserved, set to Zero |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| IPVS Sync Connection (1) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| . |
~ . ~
| . |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| IPVS Sync Connection (n) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version 0 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Count Conns | SyncID | Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPVS Sync Connection (1) |

The Sync. Conn. message format is also changed to allow future enhancements and versions.

Sync Connection format (sync_conn)

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Protocol | Ver. | Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| State | cport |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| vport | dport |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fwmark |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timeout (in sec.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
| IP-Addresses (v4 or v6) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Optional Parameters.

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Param. Type    | Param. Length |   Param. data                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                              ...                              |
      |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               | Param Type    | Param. Length |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           Param  data                         |
      |         Last Param data should be padded for 32 bit alignment |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

New fields that might need some explanation,

Type:
Bit 7 6 . . . 1 0

    +--------------------------------------------------+-------+
    |                      Spare                       |  IPv6 |
    +--------------------------------------------------+-------+

Bit 0 = 1 This sync_conn contains IPv6 addresses.

Ver. Version of specified type right now it's 0

fwmark: Firewall mark from skb.

timeout: from ip_vs_conn struct, converted to sec.

Optional Parameters will be added at the end without any kind of alignment each Param will have a header of two bytes

Param. header
Bit 7 6 . . . 0 7 0

   +----------+----------------------+---------------------------+
   | Optional |  Parameter Type      |  Parameter length         |
   +----------+----------------------+---------------------------+
   |  Data ...                                                   |
   |  ...                                                        |
   +----------+----------------------+---------------------------+

Bit 7 (msb) Optional Param, conn. entry could be keept, if not type
is unknown.
Bit 6-0 Option Type defined below

#define IPVS_OPT_SEQ_DATA       1
#define IPVS_OPT_PE_DATA        2
#define IPVS_OPT_PE_NAME        3

Option Length, 1-255 bytes 0 is an illegal value.

PATCH STATUS:
- Persistence data is not tested.

THANKS
To Julian and Simon for the review and alternative solutions.

*v3
New: Ability to send ver. 0 sync messages.
Optional data, renamed to Optional parameter
Corrections, see comments in each patch,

*v2
Simplified fwmark handling patch 1/4
New handling of optional data patch 2/4
Seconds in timeout
Basically all changes is based on Julians and Simons comments on the RFC For details see individual patches.
--
Regards
Hans Schillstrom <[email protected]>

maintainer
#1 lore raw

Hello,

Quoted 15 lines · Hans Schillstrom

This patch series adds/(updates) the following functionality in the synchronization between master and backup daemons.

- IPv6
- Persistence Engine
- Firewall marks transferred
- Time-outs transferred.
- Flag field increased to 32 bits.

Note:
This patches depends on Patch 1-9 in staging branch:
"IPVS: Backup, Prepare for transferring firewall marks (fwmark) to the backup daemon." to ...
"IPVS: skb defrag in L7 helpers"

Quoted 5 lines

A new message format is also introduced, not understood by old backup daemons. For compatibility reasons receiving the old version (version 0) is still possible. Old (version 0) backups will just drop new (Version 1) messages.
It's also possible to send sync msg in version 0 format, by sysctl #sysclt -w net.ipv4.vs.sync_version=0

v3 PATCH 1/3

	- ntoh_seq can use directly get_unaligned_be32, it is BE to CPU,
		no need for ntohl

	- In ip_vs_conn_fill_param_sync:

		- should we ignore the conn entry when pe_name_len
		is 0 and pe_data_len != 0 (return 1)? May be such checks:

		/* Handle pe data */
		if ((pe_data_len != 0) != (pe_name_len != 0)) {
			IP_VS_DBG(3, "BACKUP, Invalid PE parameters\n");
			return 1;
		}
		if (!pe_name_len)
			return 0;
		...

		- make sure p->pe is _put when p->pe_data
			allocation fails (-ENOMEM)

	- ip_vs_proc_str: use 'unsigned int', not just 'unsigned'

	- ip_vs_proc_sync_conn:
		- pe_data_len, pe_name_len, pe_data, pe_name are
		not initialized

		- 'retc 10;' must be 'retc = 10;' ?

		- inconsistent spaces:
			'} else if (! s->v4.type) {'
			'while (p< msg_end && *p && !retc ) {'

		- please convert char *p, char *msg_end to '__u8 *' or
		'unsigned char *' because this is risky for char * when
		value is above 127:
			int ptype = *(p++);
			int plen =  *(p++);

		- add some sanity checks because 'p< msg_end' guarantees
		that only ptype is present, for example:

			while (p < msg_end) {
				int ptype;
				int plen;

				if (p + 2 > msg_end)
					return -30;

				ptype = *(p++);
				plen = *(p++);

		- for IPVS_OPT_PE_NAME ip_vs_proc_str must be called
		with IPVS_OPT_F_PE_NAME

	- ip_vs_process_message:
		- before 'size = ntohs(s->v4.ver_size) & SVER_MASK;'
		there must be a check for present v4 structure:

		if (msg_end + sizeof(s->v4) > buffer+buflen) {
			IP_VS_ERR_RL("BACKUP, Dropping buffer, msg > buffer\n");
			return;
		}
		size = ntohs(s->v4.ver_size) & SVER_MASK;

		- we must account padding later after
		ip_vs_proc_sync_conn is called, it should not be
		included in size because the parameter
		parsing will fail while walking the padding:

			size += 3;
			size &= ~3;
			msg_end = p + size;

		there is no need to check if new msg_end
		exceeds buflen because if there are more
		conn entries the above code will check for
		present v4 structure.

		Another option is the parameter parsing
		to consider ptype=0 as 1 byte padding but
		then there should be no plen octet for this
		case. Then ver_size can include the padding.
		For example:

		while (p < msg_end) {
			int ptype = *(p++);
			int plen;

			/* Padding ? */
			if (!ptype)
				continue;
			if (p + 1 > msg_end)
				return -30;
			plen = *(p++);

v3 PATCH 2/3

	- hton_seq can use put_unaligned_be32
	- ip_vs_sync_conn:
		- move init of pe_name_len after sloop

		- The 'Sanity checks' code must be after sloop

		- we can use cp->pe instead of cp->dest->svc->pe

		- before 'len = (len+3) & 0xffc;' may be
		we have to calculate pad = (4 - len) & 3
		so that we can use it later, len should not
		include the padding, for example:

		pad = (4 - len) & 3;
		/* check if there is a space for this one */
		if (curr_sb && (curr_sb->head+len+pad > curr_sb->end)) {
		...

		- then after all parameters:

		curr_sb->head += len;
		if (pad) {
			memset(curr_sb->head, 0, pad);
			curr_sb->head += pad;
		}

		or

		curr_sb->head += len;
		while (pad --)
			*curr_sb->head ++ = 0;

		then this code is not needed:

		+	if (p < curr_sb->head)
		+		*p = 0; 	/* Dont leave random bytes at end */

		- use cp->pe for IPVS_OPT_PE_NAME

v3 PATCH 3/3
OK

Regards

--
Julian Anastasov <[email protected]>

#2 lore raw
Quoted 27 lines · On Saturday 13 November 2010 00:31:21 you wrote:

Hello,

Quoted 15 lines · Hans Schillstrom

This patch series adds/(updates) the following functionality in the synchronization between master and backup daemons.

- IPv6
- Persistence Engine
- Firewall marks transferred
- Time-outs transferred.
- Flag field increased to 32 bits.

Note:
This patches depends on Patch 1-9 in staging branch:
"IPVS: Backup, Prepare for transferring firewall marks (fwmark) to the backup daemon." to ...
"IPVS: skb defrag in L7 helpers"

Quoted 5 lines

A new message format is also introduced, not understood by old backup daemons. For compatibility reasons receiving the old version (version 0) is still possible. Old (version 0) backups will just drop new (Version 1) messages.
It's also possible to send sync msg in version 0 format, by sysctl #sysclt -w net.ipv4.vs.sync_version=0

v3 PATCH 1/3

Review

Quoted 2 lines
(quoted diff) +0 -0
 	- ntoh_seq can use directly get_unaligned_be32, it is BE to CPU,
 		no need for ntohl

Tanks,

Quoted 4 lines
(quoted diff) +0 -0
 	- In ip_vs_conn_fill_param_sync:
 
 		- should we ignore the conn entry when pe_name_len

Review

Quoted 10 lines
(quoted diff) +0 -0
 		is 0 and pe_data_len != 0 (return 1)? May be such checks:
 
 		/* Handle pe data */
 		if ((pe_data_len != 0) != (pe_name_len != 0)) {
 			IP_VS_DBG(3, "BACKUP, Invalid PE parameters\n");
 			return 1;
 		}
 		if (!pe_name_len)
 			return 0;
...

It's OK, for me, what do you think Simon ?
Lets do it this way,

	if (pe_data_len) {
		if (pe_name_len) {
			...
		} else {
			IP_VS_DBG(3, "BACKUP, Invalid PE parameters\n");
			return 1;
		}
		...

Review

Quoted 4 lines
(quoted diff) +0 -0
 		- make sure p->pe is _put when p->pe_data
 			allocation fails (-ENOMEM)
		if (!p->pe_data) {
			if (p->pe->module)
				module_put(p->pe->module);
			return -ENOMEM;
		}
Quoted 2 lines

- ip_vs_proc_str: use 'unsigned int', not just 'unsigned'

OK

Review

Quoted 3 lines
(quoted diff) +0 -0
 	- ip_vs_proc_sync_conn:
 		- pe_data_len, pe_name_len, pe_data, pe_name are
 		not initialized

Ooops, how did that happen :-?

Quoted 2 lines

- 'retc 10;' must be 'retc = 10;' ?

Yes it should, I wounder why the compiler didn't complain...

Review

Quoted 4 lines
(quoted diff) +0 -0
 		- inconsistent spaces:
 			'} else if (! s->v4.type) {'
 			'while (p< msg_end && *p && !retc ) {'

My fingers seems to like the space bar :-)

Review

Quoted 7 lines
(quoted diff) +0 -0
 		- please convert char *p, char *msg_end to '__u8 *' or
 		'unsigned char *' because this is risky for char * when
 		value is above 127:
 			int ptype = *(p++);
 			int plen =  *(p++);

OK, I switch to __u8 in general;

Review

Quoted 2 lines
(quoted diff) +0 -0
 		- add some sanity checks because 'p< msg_end' guarantees
 		that only ptype is present, for example:

Thats true

Review

Quoted 13 lines
(quoted diff) +0 -0
 			while (p < msg_end) {
 				int ptype;
 				int plen;
 
 				if (p + 2 > msg_end)
 					return -30;
 
 				ptype = *(p++);
 				plen = *(p++);
 
 		- for IPVS_OPT_PE_NAME ip_vs_proc_str must be called
 		with IPVS_OPT_F_PE_NAME

Oops

Review

Quoted 9 lines
(quoted diff) +0 -0
 	- ip_vs_process_message:
 		- before 'size = ntohs(s->v4.ver_size) & SVER_MASK;'
 		there must be a check for present v4 structure:
 
 		if (msg_end + sizeof(s->v4) > buffer+buflen) {
 			IP_VS_ERR_RL("BACKUP, Dropping buffer, msg > buffer\n");
 			return;
 		}

OK

Quoted 5 lines
(quoted diff) +0 -0
 		size = ntohs(s->v4.ver_size) & SVER_MASK;
 
 		- we must account padding later after
 		ip_vs_proc_sync_conn is called, it should not be
 		included in size because the parameter

Review

Quoted 10 lines
(quoted diff) +0 -0
 		parsing will fail while walking the padding:
 
 			size += 3;
 			size &= ~3;
 			msg_end = p + size;
 
 		there is no need to check if new msg_end
 		exceeds buflen because if there are more
 		conn entries the above code will check for
 		present v4 structure.

I think this one produce cleaner code,

Quoted 18 lines
(quoted diff) +0 -0
 		Another option is the parameter parsing
 		to consider ptype=0 as 1 byte padding but
 		then there should be no plen octet for this
 		case. Then ver_size can include the padding.
 		For example:
 
 		while (p < msg_end) {
 			int ptype = *(p++);
 			int plen;
 
 			/* Padding ? */
 			if (!ptype)
 				continue;
 			if (p + 1 > msg_end)
 				return -30;
 			plen = *(p++);
Quoted 2 lines

v3 PATCH 2/3
- hton_seq can use put_unaligned_be32

Yes

Review

Quoted 3 lines
(quoted diff) +0 -0
 	- ip_vs_sync_conn:
 		- move init of pe_name_len after sloop

Ooops

Quoted 2 lines

- The 'Sanity checks' code must be after sloop

Yes,

Quoted 1 line

- we can use cp->pe instead of cp->dest->svc->pe

OK

Quoted 41 lines
(quoted diff) +0 -0
 		- before 'len = (len+3) & 0xffc;' may be
 		we have to calculate pad = (4 - len) & 3
 		so that we can use it later, len should not
 		include the padding, for example:
 
 
 		pad = (4 - len) & 3;
 		/* check if there is a space for this one */
 		if (curr_sb && (curr_sb->head+len+pad > curr_sb->end)) {
...
 
 		- then after all parameters:
 
 		curr_sb->head += len;
 		if (pad) {
 			memset(curr_sb->head, 0, pad);
 			curr_sb->head += pad;
 		}
 
 		or
 
 		curr_sb->head += len;
 		while (pad --)
 			*curr_sb->head ++ = 0;
 
 		then this code is not needed:
 
 		+	if (p < curr_sb->head)
 		+		*p = 0; 	/* Dont leave random bytes at end */
 
 		- use cp->pe for IPVS_OPT_PE_NAME

v3 PATCH 3/3
OK

Regards

--
Julian Anastasov <[email protected]>

Signature · 2 lines
Regards
Hans Schillstrom <[email protected]>
maintainer
#3 lore raw
Quoted 59 lines · Hans Schillstrom
Quoted 27 lines · On Saturday 13 November 2010 00:31:21 you wrote:

Hello,

Quoted 15 lines · Hans Schillstrom

This patch series adds/(updates) the following functionality in the synchronization between master and backup daemons.

- IPv6
- Persistence Engine
- Firewall marks transferred
- Time-outs transferred.
- Flag field increased to 32 bits.

Note:
This patches depends on Patch 1-9 in staging branch:
"IPVS: Backup, Prepare for transferring firewall marks (fwmark) to the backup daemon." to ...
"IPVS: skb defrag in L7 helpers"

Quoted 5 lines

A new message format is also introduced, not understood by old backup daemons. For compatibility reasons receiving the old version (version 0) is still possible. Old (version 0) backups will just drop new (Version 1) messages.
It's also possible to send sync msg in version 0 format, by sysctl #sysclt -w net.ipv4.vs.sync_version=0

v3 PATCH 1/3

Review

Quoted 2 lines
(quoted diff) +0 -0
 	- ntoh_seq can use directly get_unaligned_be32, it is BE to CPU,
 		no need for ntohl

Tanks,

Quoted 4 lines
(quoted diff) +0 -0
 	- In ip_vs_conn_fill_param_sync:
 
 		- should we ignore the conn entry when pe_name_len

Review

Quoted 10 lines
(quoted diff) +0 -0
 		is 0 and pe_data_len != 0 (return 1)? May be such checks:
 
 		/* Handle pe data */
 		if ((pe_data_len != 0) != (pe_name_len != 0)) {
 			IP_VS_DBG(3, "BACKUP, Invalid PE parameters\n");
 			return 1;
 		}
 		if (!pe_name_len)
 			return 0;
...

It's OK, for me, what do you think Simon ?
Lets do it this way,

	if (pe_data_len) {
		if (pe_name_len) {
			...
		} else {
			IP_VS_DBG(3, "BACKUP, Invalid PE parameters\n");
			return 1;
		}
		...
Quoted 1 line
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +0 -0
 		- make sure p->pe is _put when p->pe_data
 			allocation fails (-ENOMEM)
		if (!p->pe_data) {
			if (p->pe->module)
				module_put(p->pe->module);
			return -ENOMEM;
		}

I think that looks reasonable