Skip to contentSkip to conversation

Why netboot:

PATCH New DH at synoia.com 2 participants 5 messages May 20, 2018 – Jun 12, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v11 cover Why netboot:

From: Duncan Hare <[email protected]>

Central management, including logs and change control, coupled with with enhanced security and unauthorized change detection and remediation by exposing a small attack surface.

Why TCP with Selective Acknowledgement:

Currently file transfer are done using tftp or NFS both over udp. This requires a request to be sent from client (u-boot) to the boot server.

For a 4 Mbyte kernel, with a 1k block size this requires 4,000 request for a block.

Using a large block size, one greater than the Ethernet maximum frame size limitation, would require fragmentation, which u-boot supports. However missing fragment recovery requires timeout detection and re-transmission requests for missing fragments.

UDP is ideally suited to fast single packet exchanges, inquiry/response, for example dns, becuse of the lack of connection overhead.

UDP as a file transport mechanism is slow, even in low latency networks, because file transfer with udp requires poll/response mechanism to provide transfer integrity.

In networks with large latency, for example: the internet, UDP is even slower. What is a 30 second transfer on a local boot server and LAN increase to over 3 minutes, because of all the requests/response traffic.

This was anticipated in the evolution of the IP protocols and TCP was developed and then enhanced for high latency high bandwidth networks.

The current standard is TCP with selective acknowledgment.

In our testing we have reduce kernel transmission time to around 0.4 seconds for a 4Mbyte kernel, with a 100 Mbps downlink.

Why http and wget:

HTTP is the most efficient file retrieval protocol in common use. The client send a single request, after TCP connection, to receive a file of any length.

WGET is the application which implements http file transfer outside browsers as a file transfer protocol. Versions of wget exists on many operating systems.

Changes in v11:
Add TCP with Selective Acknowledgement (SACK)
Adding wget

Duncan Hare (3):
Adding TCP and wget into u-boot
Adding TCP
Add wget, for fast file transfer.

 cmd/Kconfig        |   7 +
 cmd/net.c          |  13 +
 include/net.h      |  15 +-
 include/net/tcp.h  | 227 +++++++++++++++++
 include/net/wget.h |  19 ++
 net/Kconfig        |   6 +
 net/Makefile       |   2 +
 net/net.c          |  79 +++++-
 net/ping.c         |   9 +-
 net/tcp.c          | 700 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/wget.c         | 418 ++++++++++++++++++++++++++++++++
 11 files changed, 1475 insertions(+), 20 deletions(-)
 create mode 100644 include/net/tcp.h
 create mode 100644 include/net/wget.h
 create mode 100644 net/tcp.c
 create mode 100644 net/wget.c
Signature · 1 line
2.11.0
#1 lore raw

PATCH v11 1/3 Adding TCP and wget into u-boot

From: Duncan Hare <[email protected]>

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Initial changes for adding TCP
Meet change control standards after review

<<<<

---

Consolidatind UDP header functions to make it easier to add TCP versions of the same, while reusing IP portions. This patch should not change any behaviors.

Add a protocol parameter to ip packet sending in net.c Add UDP protocol for current applications to minimize code changes to existing ping.c

All the code is new, and not copied from any source.

Changes in v11: None

 include/net.h |  7 ++++++-
 net/net.c     | 32 +++++++++++++++++++++++---------
 net/ping.c    |  9 ++-------
 3 files changed, 31 insertions(+), 17 deletions(-)
include/net.h +6 -1
index 65f51d77a5..ac80aaddd8 100644
@@ -593,7 +593,8 @@ int net_set_ether(uchar *xet, const uchar *dest_ethaddr, uint prot);
 int net_update_ether(struct ethernet_hdr *et, uchar *addr, uint prot);
 
 /* Set IP header */
-void net_set_ip_header(uchar *pkt, struct in_addr dest, struct in_addr source);
+void net_set_ip_header(uchar *pkt, struct in_addr dest, struct in_addr source,
+		       u16  pkt_len, u8 prot);
 void net_set_udp_header(uchar *pkt, struct in_addr dest, int dport,
 				int sport, int len);
 
@@ -667,6 +668,10 @@ static inline void net_send_packet(uchar *pkt, int len)
  * @param sport Source UDP port
  * @param payload_len Length of data after the UDP header
  */
+int net_send_ip_packet(uchar *ether, struct in_addr dest, int dport, int sport,
+		       int payload_len, int proto, u8 action, u32 tcp_seq_num,
+		       u32 tcp_ack_num);
+
 int net_send_udp_packet(uchar *ether, struct in_addr dest, int dport,
 			int sport, int payload_len);
 
net/net.c +23 -9
index 7f85211442..607f0fe865 100644
@@ -777,6 +777,14 @@ void net_set_timeout_handler(ulong iv, thand_f *f)
 int net_send_udp_packet(uchar *ether, struct in_addr dest, int dport, int sport,
 		int payload_len)
 {
+	return net_send_ip_packet(ether, dest, dport, sport, payload_len,
+				  IPPROTO_UDP, 0, 0, 0);
+}
+
+int net_send_ip_packet(uchar *ether, struct in_addr dest, int dport, int sport,
+		       int payload_len, int proto, u8 action, u32 tcp_seq_num,
+		       u32 tcp_ack_num)
+{
 	uchar *pkt;
 	int eth_hdr_size;
 	int pkt_hdr_size;
@@ -797,9 +805,14 @@ int net_send_udp_packet(uchar *ether, struct in_addr dest, int dport, int sport,
 	pkt = (uchar *)net_tx_packet;
 
 	eth_hdr_size = net_set_ether(pkt, ether, PROT_IP);
-	pkt += eth_hdr_size;
-	net_set_udp_header(pkt, dest, dport, sport, payload_len);
-	pkt_hdr_size = eth_hdr_size + IP_UDP_HDR_SIZE;
+	switch (proto) {
+	case IPPROTO_UDP:
+		net_set_udp_header(pkt + eth_hdr_size, dest,
+				   dport, sport, payload_len);
+		pkt_hdr_size = eth_hdr_size + IP_UDP_HDR_SIZE;
+		break;
+	default: return -EINVAL;
+	}
 
 	/* if MAC address was not discovered yet, do an ARP request */
 	if (memcmp(ether, net_null_ethaddr, 6) == 0) {
@@ -1424,7 +1437,8 @@ int net_update_ether(struct ethernet_hdr *et, uchar *addr, uint prot)
 	}
 }
 
-void net_set_ip_header(uchar *pkt, struct in_addr dest, struct in_addr source)
+void net_set_ip_header(uchar *pkt, struct in_addr dest, struct in_addr source,
+		       u16  pkt_len, u8 prot)
 {
 	struct ip_udp_hdr *ip = (struct ip_udp_hdr *)pkt;
 
@@ -1434,15 +1448,17 @@ void net_set_ip_header(uchar *pkt, struct in_addr dest, struct in_addr source)
 	/* IP_HDR_SIZE / 4 (not including UDP) */
 	ip->ip_hl_v  = 0x45;
 	ip->ip_tos   = 0;
-	ip->ip_len   = htons(IP_HDR_SIZE);
+	ip->ip_len   = htons(pkt_len);
 	ip->ip_id    = htons(net_ip_id++);
 	ip->ip_off   = htons(IP_FLAGS_DFRAG);	/* Don't fragment */
 	ip->ip_ttl   = 255;
+	ip->ip_p     = prot;
 	ip->ip_sum   = 0;
 	/* already in network byte order */
 	net_copy_ip((void *)&ip->ip_src, &source);
 	/* already in network byte order */
 	net_copy_ip((void *)&ip->ip_dst, &dest);
+	ip->ip_sum  = compute_ip_checksum(ip, IP_HDR_SIZE);
 }
 
 void net_set_udp_header(uchar *pkt, struct in_addr dest, int dport, int sport,
@@ -1458,10 +1474,8 @@ void net_set_udp_header(uchar *pkt, struct in_addr dest, int dport, int sport,
 	if (len & 1)
 		pkt[IP_UDP_HDR_SIZE + len] = 0;
 
-	net_set_ip_header(pkt, dest, net_ip);
-	ip->ip_len   = htons(IP_UDP_HDR_SIZE + len);
-	ip->ip_p     = IPPROTO_UDP;
-	ip->ip_sum   = compute_ip_checksum(ip, IP_HDR_SIZE);
+	net_set_ip_header(pkt, dest, net_ip, IP_UDP_HDR_SIZE + len,
+			  IPPROTO_UDP);
 
 	ip->udp_src  = htons(sport);
 	ip->udp_dst  = htons(dport);
net/ping.c +2 -7
index 5464f2f785..50905d2ad6 100644
@@ -20,16 +20,11 @@ struct in_addr net_ping_ip;
 static void set_icmp_header(uchar *pkt, struct in_addr dest)
 {
 	/*
-	 *	Construct an IP and ICMP header.
+	 *	Construct an ICMP header.
 	 */
-	struct ip_hdr *ip = (struct ip_hdr *)pkt;
 	struct icmp_hdr *icmp = (struct icmp_hdr *)(pkt + IP_HDR_SIZE);
 
-	net_set_ip_header(pkt, dest, net_ip);
-
-	ip->ip_len   = htons(IP_ICMP_HDR_SIZE);
-	ip->ip_p     = IPPROTO_ICMP;
-	ip->ip_sum   = compute_ip_checksum(ip, IP_HDR_SIZE);
+	net_set_ip_header(pkt, dest, net_ip, IP_ICMP_HDR_SIZE, IPPROTO_ICMP);
 
 	icmp->type = ICMP_ECHO_REQUEST;
 	icmp->code = 0;
Signature · 1 line
2.11.0
#2 lore raw

PATCH v11 2/3 Adding TCP

In reply to DH at synoia.com

From: Duncan Hare <[email protected]>

All the code is new, and not copied from any source.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

For Multi-hop fast netwoks, (aka Long Fat networks) SACK provides continued stream teansfer without delaying the transfer for a misssing packet.

<<<<

---
Clean code post review.

Changes in v11:
Add TCP with Selective Acknowledgement (SACK)

 include/net.h     |  12 +-
 include/net/tcp.h | 227 ++++++++++++++++++
 net/Kconfig       |   6 +
 net/Makefile      |   1 +
 net/tcp.c         | 700 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 943 insertions(+), 3 deletions(-)
 create mode 100644 include/net/tcp.h
 create mode 100644 net/tcp.c
include/net.h +9 -3
index ac80aaddd8..47ef673f61 100644
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: GPL-2.0 */
+// SPDX-License-Identifier: GPL-2.0
 /*
  *	LiMon Monitor (LiMon) - Network.
  *
@@ -25,7 +25,9 @@
  *	alignment in memory.
  *
  */
-
+#if defined(CONFIG_TCP)		/* Protected becuse UDP uses less buffers */
+#define CONFIG_SYS_RX_ETH_BUFFER 12
+#endif
 #ifdef CONFIG_SYS_RX_ETH_BUFFER
 # define PKTBUFSRX	CONFIG_SYS_RX_ETH_BUFFER
 #else
@@ -84,7 +86,7 @@ enum eth_state_t {
 };
 
 #ifdef CONFIG_DM_ETH
-/**
+/*
  * struct eth_pdata - Platform data for Ethernet MAC controllers
  *
  * @iobase: The base address of the hardware registers
@@ -350,6 +352,7 @@ struct vlan_ethernet_hdr {
 #define PROT_PPP_SES	0x8864		/* PPPoE session messages	*/
 
 #define IPPROTO_ICMP	 1	/* Internet Control Message Protocol	*/
+#define IPPROTO_TCP      6      /* Transmission Control Protocol        */
 #define IPPROTO_UDP	17	/* User Datagram Protocol		*/
 
 /*
@@ -675,6 +678,9 @@ int net_send_ip_packet(uchar *ether, struct in_addr dest, int dport, int sport,
 int net_send_udp_packet(uchar *ether, struct in_addr dest, int dport,
 			int sport, int payload_len);
 
+int net_send_tcp_packet(int payload_len, int dport, int sport, u8 action,
+			u32 tcp_seq_num, u32 tcp_ack_num);
+
 /* Processes a received packet */
 void net_process_received_packet(uchar *in_packet, int len);
 
include/net/tcp.h new file +227 -0

227 lines not on this page · raw message

net/Kconfig +6 -0
index f2363e5256..77ab683eb8 100644
@@ -22,4 +22,10 @@ config NETCONSOLE
 	  Support the 'nc' input/output device for networked console.
 	  See README.NetConsole for details.
 
+config TCP
+	bool "TCP stack"
+	help
+	  TCP protocol support with SACK for wget. Selecting this will provide
+	  the fastest file transfer possible.
+
 endif   # if NET
net/Makefile +1 -0
index d1e8e01f62..c2a7c2d990 100644
@@ -23,6 +23,7 @@ obj-$(CONFIG_CMD_PING) += ping.o
 obj-$(CONFIG_CMD_RARP) += rarp.o
 obj-$(CONFIG_CMD_SNTP) += sntp.o
 obj-$(CONFIG_CMD_TFTPBOOT) += tftp.o
+obj-$(CONFIG_TCP)      += tcp.o
 
 # Disable this warning as it is triggered by:
 # sprintf(buf, index ? "foo%d" : "foo", index)
net/tcp.c new file +700 -0

700 lines not on this page · raw message

Signature · 1 line
2.11.0
#3 lore raw

PATCH v11 3/3 Add wget, for fast file transfer.

In reply to DH at synoia.com

From: Duncan Hare <[email protected]>

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

<<<<

---
All the code is new, and not copied from any source.

Changes in v11:
Adding wget

 cmd/Kconfig        |   7 +
 cmd/net.c          |  13 ++
 include/net.h      |   8 +-
 include/net/wget.h |  19 +++
 net/Makefile       |   1 +
 net/net.c          |  49 ++++++-
 net/wget.c         | 418 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 508 insertions(+), 7 deletions(-)
 create mode 100644 include/net/wget.h
 create mode 100644 net/wget.c
cmd/Kconfig +7 -0
index eb14e08f77..9a76f42377 100644
@@ -1156,6 +1156,13 @@ config CMD_NFS
 	help
 	  Boot image via network using NFS protocol.
 
+config CMD_WGET
+        bool "wget"
+        select TCP
+        help
+          Download kernel, or other files, from a web server over TCP.
+	  Fast file transfer over networks with latency.
+
 config CMD_MII
 	bool "mii"
 	help
cmd/net.c +13 -0
index f83839c35e..f5fde849c4 100644
@@ -113,6 +113,19 @@ U_BOOT_CMD(
 );
 #endif
 
+#if defined(CONFIG_CMD_WGET)
+static int do_wget(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
+{
+	return netboot_common(WGET, cmdtp, argc, argv);
+}
+
+U_BOOT_CMD(
+	wget,   3,      1,      do_wget,
+	"boot image via network using HTTP protocol",
+	"[loadAddress] [[hostIPaddr:]path and image name]"
+);
+#endif
+
 static void netboot_update_env(void)
 {
 	char tmp[22];
include/net.h +3 -5
index 47ef673f61..d94de7c62f 100644
@@ -25,9 +25,7 @@
  *	alignment in memory.
  *
  */
-#if defined(CONFIG_TCP)		/* Protected becuse UDP uses less buffers */
-#define CONFIG_SYS_RX_ETH_BUFFER 12
-#endif
+
 #ifdef CONFIG_SYS_RX_ETH_BUFFER
 # define PKTBUFSRX	CONFIG_SYS_RX_ETH_BUFFER
 #else
@@ -86,7 +84,7 @@ enum eth_state_t {
 };
 
 #ifdef CONFIG_DM_ETH
-/*
+/**
  * struct eth_pdata - Platform data for Ethernet MAC controllers
  *
  * @iobase: The base address of the hardware registers
@@ -538,7 +536,7 @@ extern int		net_restart_wrap;	/* Tried all network devices */
 
 enum proto_t {
 	BOOTP, RARP, ARP, TFTPGET, DHCP, PING, DNS, NFS, CDP, NETCONS, SNTP,
-	TFTPSRV, TFTPPUT, LINKLOCAL
+	TFTPSRV, TFTPPUT, LINKLOCAL, WGET
 };
 
 extern char	net_boot_file_name[1024];/* Boot File name */
include/net/wget.h new file +19 -0
new file mode 100644
index 0000000000..76d377fcd8
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Duncan Hare Copyright 2017
+ */
+
+void wget_start(void);			/* Begin wget */
+
+enum WGET_STATE {
+	WGET_CLOSED,
+	WGET_CONNECTING,
+	WGET_CONNECTED,
+	WGET_TRANSFERRING,
+	WGET_TRANSFERRED
+};
+
+#define	DEBUG_WGET		0	/* Set to 1 for debug messges */
+#define	SERVER_PORT		80
+#define	WGET_RETRY_COUNT	30
+#define	WGET_TIMEOUT		2000UL
net/Makefile +1 -0
index c2a7c2d990..882af145aa 100644
@@ -24,6 +24,7 @@ obj-$(CONFIG_CMD_RARP) += rarp.o
 obj-$(CONFIG_CMD_SNTP) += sntp.o
 obj-$(CONFIG_CMD_TFTPBOOT) += tftp.o
 obj-$(CONFIG_TCP)      += tcp.o
+obj-$(CONFIG_CMD_WGET) += wget.o
 
 # Disable this warning as it is triggered by:
 # sprintf(buf, index ? "foo%d" : "foo", index)
net/net.c +47 -2

97 lines not on this page · raw message

net/wget.c new file +418 -0

418 lines not on this page · raw message

Signature · 1 line
2.11.0
#4 lore raw

v11 cover Why netboot:

In reply to DH at synoia.com

Hi Duncan,

I tried a test of applying this series and the first patch doesn't apply cleanly and checkpatch complains about Missing or malformed SPDX-License-Identifier.

Can you rebase on top of the current upstream master and resend?

Quoted 1 line · On Sun, May 20, 2018 at 6:41 PM, <[email protected]> wrote:

From: Duncan Hare <[email protected]>

Is there a reason you are not using (what appears to be) your company email address in your git configuration?

Quoted 87 lines

Central management, including logs and change control, coupled with with enhanced security and unauthorized change detection and remediation by exposing a small attack surface.

Why TCP with Selective Acknowledgement:

Currently file transfer are done using tftp or NFS both over udp. This requires a request to be sent from client (u-boot) to the boot server.

For a 4 Mbyte kernel, with a 1k block size this requires 4,000 request for a block.

Using a large block size, one greater than the Ethernet maximum frame size limitation, would require fragmentation, which u-boot supports. However missing fragment recovery requires timeout detection and re-transmission requests for missing fragments.

UDP is ideally suited to fast single packet exchanges, inquiry/response, for example dns, becuse of the lack of connection overhead.

UDP as a file transport mechanism is slow, even in low latency networks, because file transfer with udp requires poll/response mechanism to provide transfer integrity.

In networks with large latency, for example: the internet, UDP is even slower. What is a 30 second transfer on a local boot server and LAN increase to over 3 minutes, because of all the requests/response traffic.

This was anticipated in the evolution of the IP protocols and TCP was developed and then enhanced for high latency high bandwidth networks.

The current standard is TCP with selective acknowledgment.

In our testing we have reduce kernel transmission time to around 0.4 seconds for a 4Mbyte kernel, with a 100 Mbps downlink.

Why http and wget:

HTTP is the most efficient file retrieval protocol in common use. The client send a single request, after TCP connection, to receive a file of any length.

WGET is the application which implements http file transfer outside browsers as a file transfer protocol. Versions of wget exists on many operating systems.

Changes in v11:
Add TCP with Selective Acknowledgement (SACK)
Adding wget

Duncan Hare (3):
Adding TCP and wget into u-boot
Adding TCP
Add wget, for fast file transfer.

 cmd/Kconfig        |   7 +
 cmd/net.c          |  13 +
 include/net.h      |  15 +-
 include/net/tcp.h  | 227 +++++++++++++++++
 include/net/wget.h |  19 ++
 net/Kconfig        |   6 +
 net/Makefile       |   2 +
 net/net.c          |  79 +++++-
 net/ping.c         |   9 +-
 net/tcp.c          | 700 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/wget.c         | 418 ++++++++++++++++++++++++++++++++
 11 files changed, 1475 insertions(+), 20 deletions(-)
 create mode 100644 include/net/tcp.h
 create mode 100644 include/net/wget.h
 create mode 100644 net/tcp.c
 create mode 100644 net/wget.c

--
2.11.0

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot