Skip to contentSkip to conversation

dfu: samsung: Commits to allow sending and storing large files via DFU

PATCH New Lukasz Majewski 3 participants 7 messages Jan 28, 2016 – Jan 29, 2016 USB

lore
Share & tools
lore
#0 lore raw

cover dfu: samsung: Commits to allow sending and storing large files via DFU

Those commits allow sending and storing large files (e.g. 26 MiB) on the Odroid XU3 board.

On purpose they are sent together to shed light on the process of tunning board to be capable of large files transfer.

I would be _REALLY_ grateful for excessive testing, since I'm not able to test this code on NAND based devices. Please support me and test this patch for potential regressions.

This change applies on the u-boot-master tree.
SHA1: aada3d062a0cacde1e783af2ac560f2a0fdf3682

Lukasz Majewski (2):
dfu: usb: f_dfu: Set deferred call for dfu_flush() function
dfu: odroid xu3: Define DFU_MANIFEST_POLL_TIMEOUT to handle large
files transmission and storage

 common/cmd_dfu.c             | 20 ++++++++++++++++++++
 drivers/usb/gadget/f_dfu.c   | 11 +++--------
 include/configs/odroid_xu3.h |  1 +
 include/dfu.h                | 25 +++++++++++++++++++++++++
 4 files changed, 49 insertions(+), 8 deletions(-)
Signature · 1 line
2.0.0.rc2
#1 lore raw

PATCH 1/2 dfu: usb: f_dfu: Set deferred call for dfu_flush() function

This patch fixes situation when one would like to write large file into medium with the file system (fat, ext4, etc).
This change sets file size limitation to the DFU internal buffer size.

Since u-boot is not supporting interrupts and seek on file systems, it becomes challenging to store large file appropriately.

To reproduce this error - create large file (around 26 MiB) and sent it to the target board.

Lets examine the flow of USB transactions:

0. DFU uses EP0 with 64B MPS [Max Packet Size]

1. Send file - OUT (PC->target) - dat_26MiB.img is sent with 4096 B transactions

2. Get status - OUT (PC->target) - wait for DFU_STATE_dfuDNLOAD_IDLE (0x05) sent

				   from target board - IN transaction
				   (target->PC)

3. The whole file content is sent to target - OUT (PC->target) with ZLP [Zero Length Packet]

Now the interesting part starts:

4. OUT (PC->target) Setup transaction (request to share DFU state)

5. IN (target->PC) - reply the current DFU state
- In the UDC driver the req->completion (with dfu_flush) is called after successful IN transfer.
- The dfu_flush() (called from req->completion callback) saves the whole file at once (u-boot doesn't support seek on fs). Such operation takes considerable time. When the file is large - e.g. 26MiB - this time may be more than 5 seconds.

6. OUT (PC->target) - ZLP, is send in the same time when dfu_flush() writes data to eMMC memory.
The dfu-util application has hard coded timeout on USB transaction completion set to 5 seconds (it uses libusb calls).

When the file to store is large (e.g. 26 MiB) the time needed to write it may excess the dfu-util timeout and following error message will be displayed:
"unable to read DFU status" on the HOST PC console.

This change is supposed to leverage DFU's part responsible for storing files on file systems. Other DFU operations - i.e. raw/partition write to NAND and eMMC should work as before.

The only functional change is the error reporting. When dfu_flush() fails the u-boot prompt will exit with error information and dfu-util application exits afterwards as well.

Test HW:
- Odroid XU3 (Exynos5433) - test with large file
- Trats (Exynos4210) - test for regression - eMMC, raw,

---
 common/cmd_dfu.c           | 20 ++++++++++++++++++++
 drivers/usb/gadget/f_dfu.c | 11 +++--------
 include/dfu.h              | 25 +++++++++++++++++++++++++
 3 files changed, 48 insertions(+), 8 deletions(-)
common/cmd_dfu.c +20 -0
index 6d95ce9..d8aae26 100644
@@ -79,6 +79,26 @@ static int do_dfu(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 		if (ctrlc())
 			goto exit;
 
+		if (dfu_get_defer_flush()) {
+			/*
+			 * Call to usb_gadget_handle_interrupts() is necessary
+			 * to act on ZLP OUT transaction from HOST PC after
+			 * transmitting the whole file.
+			 *
+			 * If this ZLP OUT packet is NAK'ed, the HOST libusb
+			 * function fails after timeout (by default it is set to
+			 * 5 seconds). In such situation the dfu-util program
+			 * exits with error message.
+			 */
+			usb_gadget_handle_interrupts(controller_index);
+			ret = dfu_flush(dfu_get_defer_flush(), NULL, 0, 0);
+			dfu_set_defer_flush(NULL);
+			if (ret) {
+				error("Deferred dfu_flush() failed!");
+				goto exit;
+			}
+		}
+
 		WATCHDOG_RESET();
 		usb_gadget_handle_interrupts(controller_index);
 	}
drivers/usb/gadget/f_dfu.c +3 -8
index 77a1567..7d88008 100644
@@ -44,6 +44,8 @@ struct f_dfu {
 	unsigned int                    poll_timeout;
 };
 
+struct dfu_entity *dfu_defer_flush;
+
 typedef int (*dfu_state_fn) (struct f_dfu *,
 			     const struct usb_ctrlrequest *,
 			     struct usb_gadget *,
@@ -167,14 +169,7 @@ static void dnload_request_complete(struct usb_ep *ep, struct usb_request *req)
 static void dnload_request_flush(struct usb_ep *ep, struct usb_request *req)
 {
 	struct f_dfu *f_dfu = req->context;
-	int ret;
-
-	ret = dfu_flush(dfu_get_entity(f_dfu->altsetting), req->buf,
-			req->length, f_dfu->blk_seq_num);
-	if (ret) {
-		f_dfu->dfu_status = DFU_STATUS_errUNKNOWN;
-		f_dfu->dfu_state = DFU_STATE_dfuERROR;
-	}
+	dfu_set_defer_flush(dfu_get_entity(f_dfu->altsetting));
 }
 
 static inline int dfu_get_manifest_timeout(struct dfu_entity *dfu)
include/dfu.h +25 -0
index 6118dc2..f39d3f1 100644
@@ -163,6 +163,31 @@ int dfu_read(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
 int dfu_write(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
 int dfu_flush(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
 
+/*
+ * dfu_defer_flush - pointer to store dfu_entity for deferred flashing.
+ *		     It should be NULL when not used.
+ */
+extern struct dfu_entity *dfu_defer_flush;
+/**
+ * dfu_get_defer_flush - get current value of dfu_defer_flush pointer
+ *
+ * @return - value of the dfu_defer_flush pointer
+ */
+static inline struct dfu_entity *dfu_get_defer_flush(void)
+{
+	return dfu_defer_flush;
+}
+
+/**
+ * dfu_set_defer_flush - set the dfu_defer_flush pointer
+ *
+ * @param dfu - pointer to the dfu_entity, which should be written
+ */
+static inline void dfu_set_defer_flush(struct dfu_entity *dfu)
+{
+	dfu_defer_flush = dfu;
+}
+
 /**
  * dfu_write_from_mem_addr - write data from memory to DFU managed medium
  *
Signature · 1 line
2.0.0.rc2
#2 lore raw

PATCH 1/2 dfu: usb: f_dfu: Set deferred call for dfu_flush() function

Quoted 3 lines · Lukasz Majewski

This patch fixes situation when one would like to write large file into medium with the file system (fat, ext4, etc).
This change sets file size limitation to the DFU internal buffer size.

I tested this on NVIDIA's Jetson TX1 using test/py's test_dfu, with existing transfer sizes (63 bytes ... 8MB) rather than attempting to do anything with large files.

#3 lore raw

PATCH 1/2 dfu: usb: f_dfu: Set deferred call for dfu_flush() function

In reply to Lukasz Majewski

Hello Lukasz,

Am 28.01.2016 um 17:46 schrieb Lukasz Majewski:

Quoted 56 lines

This patch fixes situation when one would like to write large file into medium with the file system (fat, ext4, etc).
This change sets file size limitation to the DFU internal buffer size.

Since u-boot is not supporting interrupts and seek on file systems, it becomes challenging to store large file appropriately.

To reproduce this error - create large file (around 26 MiB) and sent it to the target board.

Lets examine the flow of USB transactions:

0. DFU uses EP0 with 64B MPS [Max Packet Size]

1. Send file - OUT (PC->target) - dat_26MiB.img is sent with 4096 B transactions

2. Get status - OUT (PC->target) - wait for DFU_STATE_dfuDNLOAD_IDLE (0x05) sent

				   from target board - IN transaction
				   (target->PC)

3. The whole file content is sent to target - OUT (PC->target) with ZLP [Zero Length Packet]

Now the interesting part starts:

4. OUT (PC->target) Setup transaction (request to share DFU state)

5. IN (target->PC) - reply the current DFU state
- In the UDC driver the req->completion (with dfu_flush) is called after successful IN transfer.
- The dfu_flush() (called from req->completion callback) saves the whole file at once (u-boot doesn't support seek on fs). Such operation takes considerable time. When the file is large - e.g. 26MiB - this time may be more than 5 seconds.

6. OUT (PC->target) - ZLP, is send in the same time when dfu_flush() writes data to eMMC memory.
The dfu-util application has hard coded timeout on USB transaction completion set to 5 seconds (it uses libusb calls).

When the file to store is large (e.g. 26 MiB) the time needed to write it may excess the dfu-util timeout and following error message will be displayed:
"unable to read DFU status" on the HOST PC console.

This change is supposed to leverage DFU's part responsible for storing files on file systems. Other DFU operations - i.e. raw/partition write to NAND and eMMC should work as before.

The only functional change is the error reporting. When dfu_flush() fails the u-boot prompt will exit with error information and dfu-util application exits afterwards as well.

Test HW:
- Odroid XU3 (Exynos5433) - test with large file
- Trats (Exynos4210) - test for regression - eMMC, raw,

Quoted 7 lines
---
  common/cmd_dfu.c           | 20 ++++++++++++++++++++
  drivers/usb/gadget/f_dfu.c | 11 +++--------
  include/dfu.h              | 25 +++++++++++++++++++++++++
  3 files changed, 48 insertions(+), 8 deletions(-)

Tested on the dxr2 board with an etamin module containing a 4GiB DDP nand with a download size from 400MiB ... worked.
(This test is not yet in my nightly builds ...)

bye,
Heiko
BTW:
I used not the "dfu_nand" module, instead I am just developing a "dfu_mtd" for accessing mtd partitions. So mtd partitions with underlying mtd concatenated nand devices for example can be programmed with dfu ... seems first version works now ...

Quoted 98 lines
common/cmd_dfu.c +20 -0
index 6d95ce9..d8aae26 100644
@@ -79,6 +79,26 @@ static int do_dfu(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
  		if (ctrlc())
  			goto exit;
 
+		if (dfu_get_defer_flush()) {
+			/*
+			 * Call to usb_gadget_handle_interrupts() is necessary
+			 * to act on ZLP OUT transaction from HOST PC after
+			 * transmitting the whole file.
+			 *
+			 * If this ZLP OUT packet is NAK'ed, the HOST libusb
+			 * function fails after timeout (by default it is set to
+			 * 5 seconds). In such situation the dfu-util program
+			 * exits with error message.
+			 */
+			usb_gadget_handle_interrupts(controller_index);
+			ret = dfu_flush(dfu_get_defer_flush(), NULL, 0, 0);
+			dfu_set_defer_flush(NULL);
+			if (ret) {
+				error("Deferred dfu_flush() failed!");
+				goto exit;
+			}
+		}
+
  		WATCHDOG_RESET();
  		usb_gadget_handle_interrupts(controller_index);
  	}
drivers/usb/gadget/f_dfu.c +3 -8
index 77a1567..7d88008 100644
@@ -44,6 +44,8 @@ struct f_dfu {
  	unsigned int                    poll_timeout;
  };
 
+struct dfu_entity *dfu_defer_flush;
+
  typedef int (*dfu_state_fn) (struct f_dfu *,
  			     const struct usb_ctrlrequest *,
  			     struct usb_gadget *,
@@ -167,14 +169,7 @@ static void dnload_request_complete(struct usb_ep *ep, struct usb_request *req)
  static void dnload_request_flush(struct usb_ep *ep, struct usb_request *req)
  {
  	struct f_dfu *f_dfu = req->context;
-	int ret;
-
-	ret = dfu_flush(dfu_get_entity(f_dfu->altsetting), req->buf,
-			req->length, f_dfu->blk_seq_num);
-	if (ret) {
-		f_dfu->dfu_status = DFU_STATUS_errUNKNOWN;
-		f_dfu->dfu_state = DFU_STATE_dfuERROR;
-	}
+	dfu_set_defer_flush(dfu_get_entity(f_dfu->altsetting));
  }
 
  static inline int dfu_get_manifest_timeout(struct dfu_entity *dfu)
include/dfu.h +25 -0
index 6118dc2..f39d3f1 100644
@@ -163,6 +163,31 @@ int dfu_read(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
  int dfu_write(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
  int dfu_flush(struct dfu_entity *de, void *buf, int size, int blk_seq_num);
 
+/*
+ * dfu_defer_flush - pointer to store dfu_entity for deferred flashing.
+ *		     It should be NULL when not used.
+ */
+extern struct dfu_entity *dfu_defer_flush;
+/**
+ * dfu_get_defer_flush - get current value of dfu_defer_flush pointer
+ *
+ * @return - value of the dfu_defer_flush pointer
+ */
+static inline struct dfu_entity *dfu_get_defer_flush(void)
+{
+	return dfu_defer_flush;
+}
+
+/**
+ * dfu_set_defer_flush - set the dfu_defer_flush pointer
+ *
+ * @param dfu - pointer to the dfu_entity, which should be written
+ */
+static inline void dfu_set_defer_flush(struct dfu_entity *dfu)
+{
+	dfu_defer_flush = dfu;
+}
+
  /**
   * dfu_write_from_mem_addr - write data from memory to DFU managed medium
   *
Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#4 lore raw

PATCH 1/2 dfu: usb: f_dfu: Set deferred call for dfu_flush() function

Hi Heiko,

Quoted 71 lines

Hello Lukasz,

Am 28.01.2016 um 17:46 schrieb Lukasz Majewski:

Quoted 60 lines

This patch fixes situation when one would like to write large file into medium with the file system (fat, ext4, etc).
This change sets file size limitation to the DFU internal buffer size.

Since u-boot is not supporting interrupts and seek on file systems, it becomes challenging to store large file appropriately.

To reproduce this error - create large file (around 26 MiB) and sent it to the target board.

Lets examine the flow of USB transactions:

0. DFU uses EP0 with 64B MPS [Max Packet Size]

1. Send file - OUT (PC->target) - dat_26MiB.img is sent with 4096 B transactions

2. Get status - OUT (PC->target) - wait for DFU_STATE_dfuDNLOAD_IDLE (0x05) sent from target board - IN transaction (target->PC)

3. The whole file content is sent to target - OUT (PC->target) with ZLP [Zero Length Packet]

Now the interesting part starts:

4. OUT (PC->target) Setup transaction (request to share DFU state)

5. IN (target->PC) - reply the current DFU state
- In the UDC driver the req->completion (with dfu_flush) is
called after successful IN transfer.
- The dfu_flush() (called from req->completion callback)
saves the whole file at once (u-boot doesn't support seek on fs).
Such operation takes considerable time. When the file is large - e.g. 26MiB - this time may be more than 5
seconds.

6. OUT (PC->target) - ZLP, is send in the same time when dfu_flush() writes data to eMMC memory.
The dfu-util application has hard coded timeout on USB transaction completion set to 5 seconds (it uses libusb calls).

When the file to store is large (e.g. 26 MiB) the time needed to write it may excess the dfu-util timeout and following error message will be displayed: "unable to read DFU status" on the HOST PC console.

This change is supposed to leverage DFU's part responsible for storing files on file systems. Other DFU operations - i.e. raw/partition write to NAND and eMMC should work as before.

The only functional change is the error reporting. When dfu_flush() fails the u-boot prompt will exit with error information and dfu-util application exits afterwards as well.

Test HW:
- Odroid XU3 (Exynos5433) - test with large file
- Trats (Exynos4210) - test for regression - eMMC, raw,

Quoted 7 lines
---
  common/cmd_dfu.c           | 20 ++++++++++++++++++++
  drivers/usb/gadget/f_dfu.c | 11 +++--------
  include/dfu.h              | 25 +++++++++++++++++++++++++
  3 files changed, 48 insertions(+), 8 deletions(-)
Quoted 5 lines

Tested on the dxr2 board with an etamin module containing a 4GiB DDP nand with a download size from 400MiB ... worked.
(This test is not yet in my nightly builds ...)

Thanks for testing!

Quoted 8 lines

bye,
Heiko
BTW:
I used not the "dfu_nand" module, instead I am just developing a "dfu_mtd" for accessing mtd partitions. So mtd partitions with underlying mtd concatenated nand devices for example can be programmed with dfu ... seems first version works now ...

Very nice to hear. I'm looking forward for your patches for review.

BTW. I do plan to setup tbot for my private Beagle Bone Black in the weekend.

Quoted 112 lines
Quoted 111 lines
common/cmd_dfu.c +3 -0
index 6d95ce9..d8aae26 100644
@@ -79,6 +79,26 @@ static int do_dfu(cmd_tbl_t *cmdtp, int flag,
 int argc, char * const argv[]) if (ctrlc())
  			goto exit;
 
+		if (dfu_get_defer_flush()) {
+			/*
+			 * Call to usb_gadget_handle_interrupts()

is necessary

common/cmd_dfu.c +1 -0
+			 * to act on ZLP OUT transaction from HOST

PC after

common/cmd_dfu.c +3 -0
+			 * transmitting the whole file.
+			 *
+			 * If this ZLP OUT packet is NAK'ed, the

HOST libusb

common/cmd_dfu.c +1 -0
+			 * function fails after timeout (by

default it is set to

common/cmd_dfu.c +1 -0
+			 * 5 seconds). In such situation the

dfu-util program

common/cmd_dfu.c +3 -0
+			 * exits with error message.
+			 */
+

usb_gadget_handle_interrupts(controller_index);

common/cmd_dfu.c +1 -0
+			ret = dfu_flush(dfu_get_defer_flush(),

NULL, 0, 0);

common/cmd_dfu.c +3 -0
+			dfu_set_defer_flush(NULL);
+			if (ret) {
+				error("Deferred dfu_flush()

failed!");

common/cmd_dfu.c +4 -0
+				goto exit;
+			}
+		}
+
  		WATCHDOG_RESET();
  		usb_gadget_handle_interrupts(controller_index);
  	}
drivers/usb/gadget/f_dfu.c +2 -0
index 77a1567..7d88008 100644
@@ -44,6 +44,8 @@ struct f_dfu {
  	unsigned int                    poll_timeout;
  };
 
+struct dfu_entity *dfu_defer_flush;
+
  typedef int (*dfu_state_fn) (struct f_dfu *,
  			     const struct usb_ctrlrequest *,
  			     struct usb_gadget *,
@@ -167,14 +169,7 @@ static void dnload_request_complete(struct

usb_ep *ep, struct usb_request *req) static void dnload_request_flush(struct usb_ep *ep, struct usb_request *req) {
struct f_dfu *f_dfu = req->context;

drivers/usb/gadget/f_dfu.c +0 -3
-	int ret;
-
-	ret = dfu_flush(dfu_get_entity(f_dfu->altsetting),

req->buf,

drivers/usb/gadget/f_dfu.c +1 -5
-			req->length, f_dfu->blk_seq_num);
-	if (ret) {
-		f_dfu->dfu_status = DFU_STATUS_errUNKNOWN;
-		f_dfu->dfu_state = DFU_STATE_dfuERROR;
-	}
+	dfu_set_defer_flush(dfu_get_entity(f_dfu->altsetting));
  }
 
  static inline int dfu_get_manifest_timeout(struct dfu_entity *dfu)
include/dfu.h +0 -0
index 6118dc2..f39d3f1 100644
@@ -163,6 +163,31 @@ int dfu_read(struct dfu_entity *de, void *buf,
 int size, int blk_seq_num); int dfu_write(struct dfu_entity *de,

void *buf, int size, int blk_seq_num); int dfu_flush(struct dfu_entity *de, void *buf, int size, int blk_seq_num);

+/*
+ * dfu_defer_flush - pointer to store dfu_entity for deferred flashing.
+ * It should be NULL when not used.
+ */

include/dfu.h +3 -0
+extern struct dfu_entity *dfu_defer_flush;
+/**
+ * dfu_get_defer_flush - get current value of dfu_defer_flush

pointer
+ *
+ * @return - value of the dfu_defer_flush pointer
+ */

include/dfu.h +15 -0
+static inline struct dfu_entity *dfu_get_defer_flush(void)
+{
+	return dfu_defer_flush;
+}
+
+/**
+ * dfu_set_defer_flush - set the dfu_defer_flush pointer
+ *
+ * @param dfu - pointer to the dfu_entity, which should be written
+ */
+static inline void dfu_set_defer_flush(struct dfu_entity *dfu)
+{
+	dfu_defer_flush = dfu;
+}
+
  /**
   * dfu_write_from_mem_addr - write data from memory to DFU

managed medium *

Signature · 5 lines
Best regards,

Lukasz Majewski

Samsung R&D Institute Poland (SRPOL) | Linux Platform Group
#5 lore raw

PATCH 2/2 dfu: odroid xu3: Define DFU_MANIFEST_POLL_TIMEOUT to handle large files transmission and storage

In reply to Lukasz Majewski

As tested on the Odroid XU3, large files to be stored on the file system require considerable time to be physically written to the medium.

The default 300 ms is not enough to store large file (e.g. 26 MiB). To fix this situation the DFU_MANIFEST_POLL_TIMEOUT has been defined.

It is used to cease the communication with dfu-util and allow the target board to store the data on file system.

---
 include/configs/odroid_xu3.h | 1 +
 1 file changed, 1 insertion(+)
include/configs/odroid_xu3.h +1 -0
index 648e48b..500f0f9 100644
@@ -69,6 +69,7 @@
 #define CONFIG_CMD_DFU
 #define CONFIG_SYS_DFU_DATA_BUF_SIZE	SZ_32M
 #define DFU_DEFAULT_POLL_TIMEOUT	300
+#define DFU_MANIFEST_POLL_TIMEOUT       25000
 
 /* THOR */
 #define CONFIG_G_DNL_THOR_VENDOR_NUM	CONFIG_G_DNL_VENDOR_NUM
Signature · 1 line
2.0.0.rc2
#6 lore raw

cover dfu: samsung: Commits to allow sending and storing large files via DFU

In reply to Lukasz Majewski

Hello Lukasz,

Am 28.01.2016 um 17:46 schrieb Lukasz Majewski:

Quoted 16 lines

Those commits allow sending and storing large files (e.g. 26 MiB) on the Odroid XU3 board.

On purpose they are sent together to shed light on the process of tunning board to be capable of large files transfer.

I would be _REALLY_ grateful for excessive testing, since I'm not able to test this code on NAND based devices. Please support me and test this patch for potential regressions.

This change applies on the u-boot-master tree.
SHA1: aada3d062a0cacde1e783af2ac560f2a0fdf3682

Lukasz Majewski (2):
dfu: usb: f_dfu: Set deferred call for dfu_flush() function
dfu: odroid xu3: Define DFU_MANIFEST_POLL_TIMEOUT to handle large
files transmission and storage

Quoted 5 lines
  common/cmd_dfu.c             | 20 ++++++++++++++++++++
  drivers/usb/gadget/f_dfu.c   | 11 +++--------
  include/configs/odroid_xu3.h |  1 +
  include/dfu.h                | 25 +++++++++++++++++++++++++
  4 files changed, 49 insertions(+), 8 deletions(-)

Tested on the smartweb board (NAND):
http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/49/steps/shell/logs/tbotlog

with current U-Boot mainline.

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany