Skip to contentSkip to conversation

USB storage: corrupted data transfers

DISCUSSION Mark Lord 2 participants 4 messages Nov 10, 2007 – Nov 11, 2007

lore
Share & tools
lore

Mark Lord

author
#0 lore raw

Something may be broken in USB / usb-storage land.

I've got a 2GB USB stick here.
I want to copy it to an image file on my hard drive:

cat /dev/sdb > usbkey.image1

Make a second copy, with or without unplugging/replugging the stick:

cat /dev/sdb > usbkey.image2

After doing this, the two copies *differ*.

So I wrote a little program to compare them and determine *how* they differ, and the result is very interesting.

Periodically, or or the other file has 8192 *zero* bytes inserted
in place of it's data. Not replacing it's data, just inserted
into the stream, causing the real data (which follows) to be offset by 8192.

If one skips over the inserted 8192 bytes in the file, the data from that point onward matches the other file, until another 8192 zeros are encountered.

The total sizes of the two image files match each other, but that's probably just due to the logic in the 'cat' program.

I wonder where those extraneous pairs of zero pages are coming from?

This makes USB drives somewhat unreliable for backups and the like, which unfortunately is exactly what just about everybody uses them for.

This (so far) is with 2.6.23.1 (w/slab allocater), and 2.6.24-rc2,
on two different machines (both Intel based, but different chipset, CPU, RAM, ...).

I suspect not all brands/models of USB sticks would give the same results, but it is rather worrisome that it happens at all.

Mmmmmm ????

#1 lore raw
Quoted 11 lines · Mark Lord

Something may be broken in USB / usb-storage land.

I've got a 2GB USB stick here.
I want to copy it to an image file on my hard drive:

cat /dev/sdb > usbkey.image1

Make a second copy, with or without unplugging/replugging the stick:

cat /dev/sdb > usbkey.image2

Quoted 8 lines

After doing this, the two copies *differ*.

So I wrote a little program to compare them and determine *how* they differ, and the result is very interesting.

Periodically, or or the other file has 8192 *zero* bytes inserted
in place of it's data. Not replacing it's data, just inserted
into the stream, causing the real data (which follows) to be offset by 8192.

That's strange. It happens periodically, you say? What's the period?

Quoted 7 lines

If one skips over the inserted 8192 bytes in the file, the data from that point onward matches the other file, until another 8192 zeros are encountered.

The total sizes of the two image files match each other, but that's probably just due to the logic in the 'cat' program.

I wonder where those extraneous pairs of zero pages are coming from?

Me too. It seems rather unlikely that they are coming from the USB device or the transport, since at that level everything is addressed in terms of sector numbers. I don't see how 16 extra sectors could just get inserted. More believable would be a problem in the block layer, the filesystem, or the application.

Quoted 2 lines

This makes USB drives somewhat unreliable for backups and the like, which unfortunately is exactly what just about everybody uses them for.

Or it makes all drives somewhat unreliable, depending on just where the problem is.

Quoted 5 lines

This (so far) is with 2.6.23.1 (w/slab allocater), and 2.6.24-rc2,
on two different machines (both Intel based, but different chipset, CPU, RAM, ...).

I suspect not all brands/models of USB sticks would give the same results, but it is rather worrisome that it happens at all.

You could try doing a more specific test. Write a program to access the block device sector-by-sector and store a distinct recognizable byte pattern to each sector. (Maybe just repeat the 4-byte sector number over and over.) Then write another program to read the data back and see what bytes ended up in which sectors.

At the same time, use the usbmon facility (see Documentation/usb/usbmon.txt) to record exactly what data gets sent to/from the stick. That should at least be sufficient to determine the problem is in the stick itself, in the SCSI/USB layers, or someplace higher up.

Alan Stern

author
#2 lore raw
In reply to Mark Lord
Quoted 5 lines · Mark Lord

Something may be broken in USB / usb-storage land.

I've got a 2GB USB stick here.
I want to copy it to an image file on my hard drive:

Quoted 7 lines

cat /dev/sdb > usbkey.image1

Make a second copy, with or without unplugging/replugging the stick:

cat /dev/sdb > usbkey.image2

After doing this, the two copies *differ*.

Here's the information on the specific devices I'm trying this with.

I'm beginning to suspect a faulty stick, or maybe one that just doesn't work reliably with the way Linux accesses it (?). The type that fails (I have several of that model here) also gives some errors from "lsusb -v" (see below for details).

I have another completely different brand/model of 2GB stick here that has now been confirmed to NOT have any such issues.

Perhaps the USB experts here could comment on the differences in information they report from lsusb --> maybe it's somethere there that makes one of them fail, and the other one work?

First, the BAD one:

Bus 001 Device 084: ID 1307:0163 Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00

 bDeviceClass            0 (Defined at Interface level)
 bDeviceSubClass         0 
 bDeviceProtocol         0 
 bMaxPacketSize0        64
 idVendor           0x1307 
 idProduct          0x0163 
 bcdDevice            1.00
 iManufacturer           1 USB 2.0
 iProduct                2 Flash Disk
 iSerial                 3 465743a79ca017
 bNumConfigurations      1
 Configuration Descriptor:
   bLength                 9
   bDescriptorType         2
   wTotalLength           39
   bNumInterfaces          1
   bConfigurationValue     1
   iConfiguration          0 
   bmAttributes         0x80
     (Bus Powered)
   MaxPower               80mA
   Interface Descriptor:
     bLength                 9
     bDescriptorType         4
     bInterfaceNumber        0
     bAlternateSetting       0
     bNumEndpoints           3
     bInterfaceClass         8 Mass Storage
     bInterfaceSubClass      6 SCSI

bInterfaceProtocol 80 Bulk (Zip)
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 1
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 1
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x83 EP 3 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 8
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00

 bDeviceClass            0 (Defined at Interface level)
 bDeviceSubClass         0 
 bDeviceProtocol         0 

bMaxPacketSize0 64
bNumConfigurations 1

can't get debug descriptor: Connection timed out cannot read device status, Connection timed out (110)

****************************

Now, the GOOD one:

Bus 001 Device 087: ID 1005:b113 Apacer Technology, Inc. Handy Steno 2.0 (256MB) Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00

 bDeviceClass            0 (Defined at Interface level)
 bDeviceSubClass         0 
 bDeviceProtocol         0 
 bMaxPacketSize0        64
 idVendor           0x1005 Apacer Technology, Inc.
 idProduct          0xb113 Handy Steno 2.0 (256MB)
 bcdDevice            1.00
 iManufacturer           1         
 iProduct                2 USB FLASH DRIVE 
 iSerial                 3 1962140002E9
 bNumConfigurations      1
 Configuration Descriptor:
   bLength                 9
   bDescriptorType         2
   wTotalLength           32
   bNumInterfaces          1
   bConfigurationValue     1
   iConfiguration          0 
   bmAttributes         0x80
     (Bus Powered)
   MaxPower              200mA
   Interface Descriptor:
     bLength                 9
     bDescriptorType         4
     bInterfaceNumber        0
     bAlternateSetting       0
     bNumEndpoints           2
     bInterfaceClass         8 Mass Storage
     bInterfaceSubClass      6 SCSI

bInterfaceProtocol 80 Bulk (Zip)
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00

 bDeviceClass            0 (Defined at Interface level)
 bDeviceSubClass         0 
 bDeviceProtocol         0 

bMaxPacketSize0 64
bNumConfigurations 1
Device Status: 0x0000
(Bus Powered)

**************************

Now just a "diff -u" of the above:

good renamed from bad +0 -1
@@ -1,4 +1,4 @@
-Bus 001 Device 084: ID 1307:0163  +Bus 001 Device 087: ID 1005:b113 Apacer Technology, Inc. Handy Steno 2.0 (256MB)

Device Descriptor:
bLength 18
bDescriptorType 1
@@ -7,29 +7,29 @@
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64

-  idVendor           0x1307 -  idProduct          0x0163 +  idVendor           0x1005 Apacer Technology, Inc.
+  idProduct          0xb113 Handy Steno 2.0 (256MB)
  bcdDevice            1.00
-  iManufacturer           1 USB 2.0
-  iProduct                2 Flash Disk
-  iSerial                 3 465743a79ca017
+  iManufacturer           1         +  iProduct                2 USB FLASH DRIVE +  iSerial                 3 1962140002E9
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
-    wTotalLength           39
+    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)

- MaxPower 80mA
+ MaxPower 200mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0

-      bNumEndpoints           3
+      bNumEndpoints           2

bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk (Zip)
@@ -37,33 +37,23 @@
Endpoint Descriptor:
bLength 7
bDescriptorType 5

-        bEndpointAddress     0x01  EP 1 OUT
+        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               1
+        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
-        bEndpointAddress     0x82  EP 2 IN
+        bEndpointAddress     0x02  EP 2 OUT

bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
- bInterval 1
- Endpoint Descriptor:

-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x83  EP 3 IN
-        bmAttributes            3
-          Transfer Type            Interrupt
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0040  1x 64 bytes
-        bInterval               8
+        bInterval               0

Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
@@ -73,7 +63,5 @@
bDeviceProtocol 0
bMaxPacketSize0 64
bNumConfigurations 1
-
-can't get debug descriptor: Connection timed out -cannot read device status, Connection timed out (110) -
+Device Status: 0x0000
+ (Bus Powered)

author
#3 lore raw
Quoted 10 lines · Mark Lord
Quoted 9 lines · Mark Lord

Something may be broken in USB / usb-storage land.

I've got a 2GB USB stick here.
I want to copy it to an image file on my hard drive:

cat /dev/sdb > usbkey.image1

Make a second copy, with or without unplugging/replugging the stick:

Quoted 8 lines
Quoted 3 lines

cat /dev/sdb > usbkey.image2

After doing this, the two copies *differ*.

Here's the information on the specific devices I'm trying this with.

I'm beginning to suspect a faulty stick, or maybe one that just doesn't work reliably with the way Linux accesses it (?). The type that fails

...

Yup, bad hardware. I've got 13 of those sticks here, and 11 are fine.

Move along folks, nothing to see here.. :)