Skip to contentSkip to conversation

2.6.8.1: ip auto-config accepts wrong packages

DISCUSSION Frank Steiner 3 participants 8 messages Aug 25, 2004 – Aug 27, 2004

lore
Share & tools
lore

Frank Steiner

author
#0 lore raw

Hi,

we are running a diskless client system using kernel ip autoconfiguration. The problem is that sometimes clients accept packages that are meant for other clients.

All clients load the kernel via pxeboot. The pxelinux.cfg contains the following append line:

append ip=dhcp nfsroot=/,tcp,nfsvers...

Usually everything goes fine. But if two clients send the dhcp request in the kernel at almost the same time, sometimes they accept the answer for the other client and then two clients end up with the same IP.

E.g., we have two clients:

  host turan {
    hardware ethernet  ...:02:11:84;
    fixed-address      ....1.167;

}

  host rado {
    hardware ethernet  ...:02:11:A5;
    fixed-address      ....1.168;

}

And sometimes we see this output from the kernel on the boot screen:

--
Sending dhcp request. <6>tg3: eth0: Link is up at 100Mbps, full duplex.
tg3: eth0: FLow control is on for TX and on for RX
,.,., OK
IP-Config: Got DHCP answer from ...1.132, my address is ...1.167 --

But this *identical* output appears on *both* screens, i.e., also on the screen of the host with the MAC :11:A5 that shouldget .168 as IP. In the server log I find this:

Aug 25 10:03:26 curry dhcpd: DHCPDISCOVER from ..:11:84 via eth0
Aug 25 10:03:26 curry dhcpd: DHCPOFFER on ...1.167 to ...:11:84 via eth0
Aug 25 10:03:26 curry dhcpd: DHCPREQUEST for ...1.167 (...1.132) from ...:11:a5 via eth0: lease ....1.167 unavailable.
Aug 25 10:03:26 curry dhcpd: DHCPNAK on ...1.167 to ...:11:a5 via eth0
Aug 25 10:03:28 curry dhcpd: DHCPDISCOVER from ...:11:a5 via eth0
Aug 25 10:03:28 curry dhcpd: DHCPOFFER on ...1.168 to ...:11:a5 via eth0
Aug 25 10:03:28 curry dhcpd: DHCPREQUEST for ...1.168 (...1.132) from ...:11:84 via eth0: lease ...1.168 unavailable.
Aug 25 10:03:28 curry dhcpd: DHCPNAK on ...1.168 to ...:11:84 via eth0
Aug 25 10:03:28 curry dhcpd: DHCPREQUEST for ...1.168 (...1.132) from ...:11:a5 via eth0
Aug 25 10:03:28 curry dhcpd: DHCPACK on ...1.168 to ...:11:a5 via eth0

So it looks like the first client request is detected at 10:03:26 and the answer is sent, but likely the second client has also sent its request in the meantime, and now both clients accept the answer meant only for the first client.
Then, when the second request is detected at 10:03:28 and answered, again both clients answer to it.

From tcpdump on the server I can indeed see that the server gets a BOOTPREQUEST from :11:84 asking for everything, and it sends a BOOTREPLY with the IP .1.167.

Next it gets a BOOTREQUEST from the :11:a5 address asking for the usual parameters (Mask etc.), but it has option 50 set:
OPTION: 50 ( 4) Request IP address 141.84.1.167

Then the answer from the server is of course the DHCPNAK.

Thus, it seems that the clients accept any answer without checking if the packet is really for them during the IP autoconfig in the kernel. Can/should they do that?

I wasn't able to reproduce the problem from the command line using dhclient, so I'm not sure if this is a general problem with dhcp or a problem in the ip-autoconfig code of the client.

Server and Clients both run 2.6.8.1 and the dhcp server is the ISC DHCP server version 3.0.1rc12.

cu,
Frank

Signature · 4 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
#1 lore raw
Quoted 11 lines · Frank Steiner

Hi,

we are running a diskless client system using kernel ip autoconfiguration. The problem is that sometimes clients accept packages that are meant for other clients.

...

I wasn't able to reproduce the problem from the command line using dhclient, so I'm not sure if this is a general problem with dhcp or a problem in the ip-autoconfig code of the client.

Try enabling the debug messaging in the ipconfig code. Edit the file net/ipv4/ipconfig.c just below the #include's and change the line

#undef IPCONFIG_DEBUG

to

#define IPCONFIG_DEBUG

and rebuild. Post the outputs from both machines when you see this effect again.

Quoted 2 lines

Server and Clients both run 2.6.8.1 and the dhcp server is the ISC DHCP server version 3.0.1rc12.

Thanks
Mark

Quoted 2 lines

cu,
Frank

author
#2 lore raw

Hi Mark,

Quoted 5 lines · Mark Broadbent

On Wed, 2004-08-25 at 10:40, Frank Steiner wrote:

Try enabling the debug messaging in the ipconfig code. Edit the file net/ipv4/ipconfig.c just below the #include's and change the line

Quoted 8 lines

#undef IPCONFIG_DEBUG

to

#define IPCONFIG_DEBUG

and rebuild. Post the outputs from both machines when you see this effect again.

the output is exactly the same on both machines! On the one machine that went through (because it got the IP first) I got the output from dmesg, printed it and compared it to the both boot screens. And they are really identical.

I didn't see any option (or "extension") that contains the mac address. Could that be the problem? Do I have to tell the server that the mac address must be in the request-list?

cu,
Frank

IP-Config: Entered.
IP-Config: eth0 UP (able=1, xid=07196018)
Sending DHCP requests .DHCP: Sending message type 1
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
DHCP: Got message type 2
DHCP: Offered address 141.84.1.168 by server 141.84.1.132 DHCP/BOOTP: Got extension 53: 02
DHCP/BOOTP: Got extension 54: 8d 54 01 84
DHCP/BOOTP: Got extension 51: 00 00 1c 20
DHCP/BOOTP: Got extension 1: ff ff ff 80
DHCP/BOOTP: Got extension 3: 8d 54 01 82
DHCP/BOOTP: Got extension 6: 81 bb d6 87
DHCP/BOOTP: Got extension 12: 72 61 64 6f
DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65 DHCP/BOOTP: Got extension 17: 2f
DHCP/BOOTP: Got extension 28: 8d 54 01 ff
DHCP/BOOTP: Got extension 208: f1 00 74 7e
DHCP/BOOTP: Got extension 209: 70 78 65 6c 69 6e 75 78 2e 63 66 67 2e 72 61 64 6f DHCP/BOOTP: Got extension 210: 68 6f 73 74 73 2f
,DHCP: Sending message type 3
DHCP: Got message type 5
DHCP/BOOTP: Got extension 53: 05
DHCP/BOOTP: Got extension 54: 8d 54 01 84
DHCP/BOOTP: Got extension 51: 00 00 1c 20
DHCP/BOOTP: Got extension 1: ff ff ff 80
DHCP/BOOTP: Got extension 3: 8d 54 01 82
DHCP/BOOTP: Got extension 6: 81 bb d6 87
DHCP/BOOTP: Got extension 12: 72 61 64 6f
DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65 DHCP/BOOTP: Got extension 17: 2f
DHCP/BOOTP: Got extension 28: 8d 54 01 ff
DHCP/BOOTP: Got extension 208: f1 00 74 7e
DHCP/BOOTP: Got extension 209: 70 78 65 6c 69 6e 75 78 2e 63 66 67 2e 72 61 64 6f DHCP/BOOTP: Got extension 210: 68 6f 73 74 73 2f
OK
IP-Config: Got DHCP answer from xxx.xxx.1.132, my address is xxx.xxx.1.168
IP-Config: Complete:

      device=eth0, addr=xxx.xxx.1.168, mask=255.255.255.128, gw=xxx.xxx.1.130,
     host=rado, domain=xxx nis-domain=(none),
     bootserver=xxx.xxx.1.132, rootserver=xxx.xxx.1.132, rootpath=/
Signature · 4 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
#3 lore raw

Hi Frank,

Quoted 5 lines · Frank Steiner

the output is exactly the same on both machines! On the one machine that went through (because it got the IP first) I got the output from dmesg, printed it and compared it to the both boot screens. And they are really identical.

Stupid question but the MAC addresses on the cards are different aren't they? Could you also double check that the xid values differ on both machines, this is shown on the line:

IP-Config: eth0 UP (able=1, xid=07196018)

                                ^^^^^^^^

Both MAC and xid should be different.

Quoted 3 lines

I didn't see any option (or "extension") that contains the mac address. Could that be the problem? Do I have to tell the server that the mac address must be in the request-list?

The MAC address is embedded in the chaddr field of the request packets.

This trace does not contain any DHCP NACK responses that were shown in the DHCP server log (annotated in the log below). Are you sure it is behaving exactly as before (i.e. is the DHCP server showing the same messages being transmitted, including the NACKs)?

The kernel appears to be working correctly in this case.

Thanks
Mark

Quoted 3 lines

IP-Config: Entered.
IP-Config: eth0 UP (able=1, xid=07196018) Sending DHCP requests .DHCP: Sending message type 1

[DHCPDISCOVER]

Quoted 3 lines

tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
DHCP: Got message type 2

[DHCPOFFER]

Quoted 7 lines

DHCP: Offered address 141.84.1.168 by server 141.84.1.132 DHCP/BOOTP: Got extension 53: 02
DHCP/BOOTP: Got extension 54: 8d 54 01 84 DHCP/BOOTP: Got extension 51: 00 00 1c 20 DHCP/BOOTP: Got extension 1: ff ff ff 80 DHCP/BOOTP: Got extension 3: 8d 54 01 82 DHCP/BOOTP: Got extension 6: 81 bb d6 87

Quoted 8 lines

DHCP/BOOTP: Got extension 12: 72 61 64 6f
DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65 DHCP/BOOTP: Got extension 17: 2f
DHCP/BOOTP: Got extension 28: 8d 54 01 ff
DHCP/BOOTP: Got extension 208: f1 00 74 7e
DHCP/BOOTP: Got extension 209: 70 78 65 6c 69 6e 75 78 2e 63 66 67 2e 72 61 64 6f DHCP/BOOTP: Got extension 210: 68 6f 73 74 73 2f
,DHCP: Sending message type 3

[DHCPREQUEST]

Quoted 1 line

DHCP: Got message type 5

[DHCPACK]

Quoted 19 lines

DHCP/BOOTP: Got extension 53: 05
DHCP/BOOTP: Got extension 54: 8d 54 01 84
DHCP/BOOTP: Got extension 51: 00 00 1c 20
DHCP/BOOTP: Got extension 1: ff ff ff 80
DHCP/BOOTP: Got extension 3: 8d 54 01 82
DHCP/BOOTP: Got extension 6: 81 bb d6 87
DHCP/BOOTP: Got extension 12: 72 61 64 6f
DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65 DHCP/BOOTP: Got extension 17: 2f
DHCP/BOOTP: Got extension 28: 8d 54 01 ff
DHCP/BOOTP: Got extension 208: f1 00 74 7e
DHCP/BOOTP: Got extension 209: 70 78 65 6c 69 6e 75 78 2e 63 66 67 2e 72 61 64 6f DHCP/BOOTP: Got extension 210: 68 6f 73 74 73 2f
OK
IP-Config: Got DHCP answer from xxx.xxx.1.132, my address is xxx.xxx.1.168
IP-Config: Complete:

       device=eth0, addr=xxx.xxx.1.168, mask=255.255.255.128, gw=xxx.xxx.1.130,
      host=rado, domain=xxx nis-domain=(none),
      bootserver=xxx.xxx.1.132, rootserver=xxx.xxx.1.132, rootpath=/
author
#4 lore raw

Hi Mark,

Quoted 8 lines · Mark Broadbent

Stupid question but the MAC addresses on the cards are different aren't they? Could you also double check that the xid values differ on both machines, this is shown on the line:

IP-Config: eth0 UP (able=1, xid=07196018)

                                ^^^^^^^^

Both MAC and xid should be different.

obiously, the problem is here. The xid is the same on 5 different hosts I tested with the debugging-enabled kernel. Looking it ipconfig.c I now understand that with several hosts using the same xid it must go wrong. I'm currently compiling a new kernel with debugging enabled in rancom.c to see what's going wrong here. Looks like random_bytes is returning the same value on all hosts :-(

Can anyone else maybe do a quick check with "ip=dhcp" on two hosts just to see if the problem with the identical xids just show up here?

I will send the result of the debugging output from random.c.

cu,
Frank

Signature · 4 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
author
#5 lore raw
In reply to Mark Broadbent

Hi Mark,

in addition to the DEBUG flags I added two little debugging lines to make sure the right code is executed.

ipconfig.c:

			if (able & IC_BOOTP){
			  DBG(("calling get_random_btyes"));
				get_random_bytes(&d->xid, sizeof(u32));

random.c:

	void get_random_bytes(void *buf, int nbytes)
	{
	  DEBUG_ENT("get_random_bytes entered\n");
	  if (sec_random_state) {
	    DEBUG_ENT("sec_random_state\n");
	    extract_entropy(sec_random_state, (char *) buf, nbytes,
			    EXTRACT_ENTROPY_SECONDARY);
	  }

And here is the output from boot.msg:

<4>IP-Config: Entered.
<4>calling get_random_btyes<7>random: get_random_bytes entered <7>random: sec_random_state
<7>random: 0000 0000 : going to reseed secondary with 64 bits (32 of 0 requested) <7>random: 0000 0000 : trying to extract 64 bits from primary <7>random: 0000 0000 : debiting 0 bits from primary
<7>random: 0000 0000 : trying to extract 32 bits from secondary <7>random: 0000 0000 : debiting 32 bits from secondary (unlimited) <4>IP-Config: eth0 UP (able=1, xid=07196018)
<5>Sending BOOTP requests .<7>random: get_random_bytes entered <7>random: sec_random_state
<7>random: 0000 0000 : going to reseed secondary with 64 bits (32 of 0 requested) <7>random: 0000 0000 : trying to extract 64 bits from primary <7>random: 0000 0000 : debiting 0 bits from primary
<7>random: 0000 0000 : trying to extract 32 bits from secondary <7>random: 0000 0000 : debiting 32 bits from secondary (unlimited) <6>tg3: eth0: Link is up at 100 Mbps, full duplex.
<6>tg3: eth0: Flow control is on for TX and on for RX. <4>.DHCP/BOOTP: Got extension 1: ff ff ff 80
<4>DHCP/BOOTP: Got extension 3: 8d 54 01 82
<4>DHCP/BOOTP: Got extension 6: 81 bb d6 87
<4>DHCP/BOOTP: Got extension 17: 2f
<4>DHCP/BOOTP: Got extension 28: 8d 54 01 ff
<4>DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65 <4> OK
<4>IP-Config: Got BOOTP answer from 141.84.1.132, my address is 141.84.1.167
...

So it looks like the code is executed, but the xid is always the same. Just like the random module would not create a random number but a constant :-) I'm not good enough in C to really understand and further debug the code in random.c or ipconfig.c :-((

cu,
Frank

Signature · 5 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *
#6 lore raw

On Thu, 26 Aug 2004 12:31:12 +0200 Frank Steiner wrote:

| Hi Mark,
| 
| in addition to the DEBUG flags I added two little debugging lines to make
| sure the right code is executed.
| 
| ipconfig.c:
| 			if (able & IC_BOOTP){
| 			  DBG(("calling get_random_btyes"));
| 				get_random_bytes(&d->xid, sizeof(u32));
| 
| random.c:
| 
| 	void get_random_bytes(void *buf, int nbytes)
| 	{
| 	  DEBUG_ENT("get_random_bytes entered\n");
| 	  if (sec_random_state) {
| 	    DEBUG_ENT("sec_random_state\n");
| 	    extract_entropy(sec_random_state, (char *) buf, nbytes,
| 			    EXTRACT_ENTROPY_SECONDARY);
| 	  }
| 
| 
| And here is the output from boot.msg:
| 
| 
| <4>IP-Config: Entered.
| <4>calling get_random_btyes<7>random: get_random_bytes entered
| <7>random: sec_random_state
| <7>random: 0000 0000 : going to reseed secondary with 64 bits (32 of 0 requested)
| <7>random: 0000 0000 : trying to extract 64 bits from primary
| <7>random: 0000 0000 : debiting 0 bits from primary
| <7>random: 0000 0000 : trying to extract 32 bits from secondary
| <7>random: 0000 0000 : debiting 32 bits from secondary (unlimited)
| <4>IP-Config: eth0 UP (able=1, xid=07196018)
| <5>Sending BOOTP requests .<7>random: get_random_bytes entered
| <7>random: sec_random_state
| <7>random: 0000 0000 : going to reseed secondary with 64 bits (32 of 0 requested)
| <7>random: 0000 0000 : trying to extract 64 bits from primary
| <7>random: 0000 0000 : debiting 0 bits from primary
| <7>random: 0000 0000 : trying to extract 32 bits from secondary
| <7>random: 0000 0000 : debiting 32 bits from secondary (unlimited)
| <6>tg3: eth0: Link is up at 100 Mbps, full duplex.
| <6>tg3: eth0: Flow control is on for TX and on for RX.
| <4>.DHCP/BOOTP: Got extension 1: ff ff ff 80
| <4>DHCP/BOOTP: Got extension 3: 8d 54 01 82
| <4>DHCP/BOOTP: Got extension 6: 81 bb d6 87
| <4>DHCP/BOOTP: Got extension 17: 2f
| <4>DHCP/BOOTP: Got extension 28: 8d 54 01 ff
| <4>DHCP/BOOTP: Got extension 15: 62 69 6f 2e 69 66 69 2e 6c 6d 75 2e 64 65
| <4> OK
| <4>IP-Config: Got BOOTP answer from 141.84.1.132, my address is 141.84.1.167
| ...
| 
| So it looks like the code is executed, but the xid is always the
| same. Just like the random module would not create a random number
| but a constant :-) I'm not good enough in C to really understand
| and further debug the code in random.c or ipconfig.c :-((

Maybe fixed by
http://linux.bkbits.net:8080/linux-2.5/cset@412a4a00MfXRfzWB5kTFo9NXM1Q3hw?nav=index.html|ChangeSet@-7d

i.e., fix is already merged, I think.

--
~Randy

author
#7 lore raw
Quoted 4 lines · Randy.Dunlap

Maybe fixed by
http://linux.bkbits.net:8080/linux-2.5/cset@412a4a00MfXRfzWB5kTFo9NXM1Q3hw?nav=index.html|ChangeSet@-7d

i.e., fix is already merged, I think.

Yes, that's exactly describing the problem I encountered, and indeed fixes it :-) I have to admit that I didn't know this page yet... I will check there first before reporting bugs in the future!

Thanks a lot!

cu,
Frank

Signature · 4 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049