XD/smartmedia - how to implement it right?
Hi,
This thread is meant to continue discussion about XD/smartmedia support in linux.
I CC'd few peoples that participated in original discussion.
Alex Dubov attempted to write a driver for jmicron device (I have it btw), but this was rejected due to code duplication.
I successfully reverse engineered (in my free time) the XD card reader found on my laptop (ricoh based one)
I would like to know how you think to implement the subsystem support right.
I would go with mmc subsystem + FTL, but according to original thread, there are numerous problems with this approach.
Any advice is appreciated.
Best regards,
Maxim Levitsky
Quoted 12 lines · Maxim Levitsky
This thread is meant to continue discussion about XD/smartmedia support in linux.
I CC'd few peoples that participated in original discussion.
Alex Dubov attempted to write a driver for jmicron device (I have it btw), but this was rejected due to code duplication.
I successfully reverse engineered (in my free time) the XD card reader found on my laptop (ricoh based one)
Quoted 7 lines
I would like to know how you think to implement the subsystem support right.
I would go with mmc subsystem + FTL, but according to original thread, there are numerous problems with this approach.
Any advice is appreciated.
A novel approach might be to just send the code as a patch. Last time around noone seemed to have had a real look.
The "mtd uses blocking calls" argument I can help with. For some other project I've added non-blocking calls to mtd[1].
Biggest problems I see are a) the userspace interface and b) allowing both raw flash access through MTD and block device access with an FTL.
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
Signature · 3 lines
The wise man seeks everything in himself; the ignorant man tries to get everything from somebody else. -- unknown
Quoted 27 lines · Jörn Engel
Quoted 12 lines · Maxim Levitsky
This thread is meant to continue discussion about XD/smartmedia support in linux.
I CC'd few peoples that participated in original discussion.
Alex Dubov attempted to write a driver for jmicron device (I have it btw), but this was rejected due to code duplication.
I successfully reverse engineered (in my free time) the XD card reader found on my laptop (ricoh based one)
Quoted 7 lines
I would like to know how you think to implement the subsystem support right.
I would go with mmc subsystem + FTL, but according to original thread, there are numerous problems with this approach.
Any advice is appreciated.
A novel approach might be to just send the code as a patch. Last time around noone seemed to have had a real look.
The "mtd uses blocking calls" argument I can help with. For some other project I've added non-blocking calls to mtd[1].
Quoted 6 lines
Biggest problems I see are a) the userspace interface and b) allowing both raw flash access through MTD and block device access with an FTL.
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I don't have much to say right now, I will study the sources for a while, and then continue the effort.
Best regards,
Maxim Levitsky
Plan for adding XD support in mtd layer
Here is my plan for adding the XD format into MTD subsystem. Please review.
New additions are marked with '->'
Chip driver:
no problems with that, in fact ricoh controller I reverse engineered is very similar conceptually
to jmicron.
I can take original jmicron driver almost verbatim.
This will live in new sub folder of mtd and call into base XD driver
Base XD mtd driver:
* No way to tell that readsize = 512, but write & erase size isn't. Not such a big deal, since higher level FTL driver can hardcode that assumption, and access via mtdblk isn't necessary, since FTL will replace it for XD, and the raw access will be done through mtdchar.
* it is possible to export 'extra' via oob. FTL will use that, and will be available to user via char device
* No need to export CIF block, it will be available for user via char device by scanning the device (using a new utility)
-> I need new mtd_info member for at least for card identification, and will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user space
FTL driver:
I will write new XD/smartmedia FTL driver.
* Will read ID/type from device and do all the work based on that, (figure page size/zone count/block size, etc...) or will get that info from mtd driver
* Will cache read/writes (this will overlap with mtdblk)
* Will provide normal block level access to filesystems, and user. (using common block device of course)
* No debugging information or/and device specific ioctls.
Char driver:
* Already exists, and I will need to modify it, so I could read ID/media type
Also would be nice to pass info about sizes (zones/pages/blocks) to userspace
If not, I will put translation table to userspace utility based on media ID
Userspace utility:
* Will allow user to read FTL maps (based on oob), Media ID, CIF, all using char device
* Will also allow low level format.
Best regards,
Maxim Levitsky
Re: Plan for adding XD support in mtd layer
Quoted 8 lines · Maxim Levitsky
Chip driver:
no problems with that, in fact ricoh controller I reverse engineered is very similar conceptually
to jmicron.
I can take original jmicron driver almost verbatim.
This will live in new sub folder of mtd and call into base XD driver
Ok. Is the code available somewhere or can you send it? And what devices would one have to buy to test it?
Quoted 10 lines
Base XD mtd driver:
* No way to tell that readsize = 512, but write & erase size isn't. Not such a big deal, since higher level FTL driver can hardcode that assumption, and access via mtdblk isn't necessary, since FTL will replace it for XD, and the raw access will be done through mtdchar.
* it is possible to export 'extra' via oob. FTL will use that, and will be available to user via char device
Quoted 7 lines
* No need to export CIF block, it will be available for user via char device by scanning the device (using a new utility)
-> I need new
mtd_infomember for at least for card identification, and will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user space
Block and page size already exist as mtd->erasesize and mtd->writesize respectively. I'm not sure whether the zone size should become part of the mtd interface - it looks more like a property of the ftl.
Quoted 16 lines
FTL driver:
I will write new XD/smartmedia FTL driver.
* Will read ID/type from device and do all the work based on that, (figure page size/zone count/block size, etc...) or will get that info from mtd driver
* Will cache read/writes (this will overlap with mtdblk)
* Will provide normal block level access to filesystems, and user. (using common block device of course)
* No debugging information or/and device specific ioctls.
Quoted 8 lines
Char driver:
* Already exists, and I will need to modify it, so I could read ID/media type
Also would be nice to pass info about sizes (zones/pages/blocks) to userspace
If not, I will put translation table to userspace utility based on media ID
What userspace programs would need this information? I can imagine some sort of xd_format or mkxd in case the card has been corrupted or used raw. Anything else?
Quoted 3 lines
Userspace utility:
* Will allow user to read FTL maps (based on oob), Media ID, CIF, all using char device
What would this be needed for?
Quoted 1 line
* Will also allow low level format.
Jörn
Signature · 3 lines
Beware of bugs in the above code; I have only proved it correct, but not tried it. -- Donald Knuth
Re: Plan for adding XD support in mtd layer
Quoted 4 lines · Jörn Engel
Quoted 3 lines · Maxim Levitsky
Chip driver:
Quoted 8 lines
Quoted 5 lines
no problems with that, in fact ricoh controller I reverse engineered is very similar conceptually
to jmicron.
I can take original jmicron driver almost verbatim.
This will live in new sub folder of mtd and call into base XD driverOk. Is the code available somewhere or can you send it? And what devices would one have to buy to test it?
Currently jmicron support is found in aspire one.
My acer has ricoh device I know how works.
Alex dubov said that TI has interest in a driver.
As soon as core is written and accepted, writing drivers will be very easy if one has documentation.
I also plan to write a 'dummy' driver to test subsystem against.
Other that work of Alex Dubov, I need to write the code, but I don't want to repeat his mistake and write code that isn't accepted on basis of lack for fitness in existing subsystems.
Quoted 15 lines
Quoted 15 lines
Base XD mtd driver:
* No way to tell that readsize = 512, but write & erase size isn't. Not such a big deal, since higher level FTL driver can hardcode that assumption, and access via mtdblk isn't necessary, since FTL will replace it for XD, and the raw access will be done through mtdchar.
* it is possible to export 'extra' via oob. FTL will use that, and will be available to user via char device
* No need to export CIF block, it will be available for user via char device by scanning the device (using a new utility)
Quoted 7 lines
Quoted 3 lines
-> I need new
mtd_infomember for at least for card identification, and will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user spaceBlock and page size already exist as
mtd->erasesizeandmtd->writesizerespectively. I'm not sure whether the zone size should become part of the mtd interface - it looks more like a property of the ftl.
You didn't understand me.
I can't nether write nor erase individual blocks (writing is technically possible, but is disallowed by recent XD spec (this is what Alex Dubov says)
However I can _read_ individual sectors
sector is 512 bytes, block is usually around 32 sectors.
Quoted 26 lines
Quoted 17 lines
FTL driver:
I will write new XD/smartmedia FTL driver.
* Will read ID/type from device and do all the work based on that, (figure page size/zone count/block size, etc...) or will get that info from mtd driver
* Will cache read/writes (this will overlap with mtdblk)
* Will provide normal block level access to filesystems, and user. (using common block device of course)
* No debugging information or/and device specific ioctls.
Quoted 8 lines
Char driver:
* Already exists, and I will need to modify it, so I could read ID/media type
Also would be nice to pass info about sizes (zones/pages/blocks) to userspace
If not, I will put translation table to userspace utility based on media ID
Quoted 3 lines
What userspace programs would need this information? I can imagine some sort of
xd_formator mkxd in case the card has been corrupted or used raw. Anything else?
I can imagine a format application, a rescue application, and an debug application that will show how healthy is the card.
There are actually two pieces of the data.
One is an ID, a serial number that can be used for example by udev for permanent mapping.
Other is the media type, and this has to be used to know the sizes of sectors, zones, pages, how many spares there are etc...
Quoted 6 lines
Quoted 4 lines
Userspace utility:
* Will allow user to read FTL maps (based on oob), Media ID, CIF, all using char deviceWhat would this be needed for?
A rescue operation, of just for curiosity...
Also to edit CIF to bypass vendor lock-in as explained in Wikipedia article
Access via mtdblock can be used to use the card as raw device too.
ECC corecting will be done in the base mtd driver, but userspace application will be able to determine if block is good or bad. If it is good, it can use oob information to determine of ECC correction was applied or not, and report overall health of the card.
Thus I think that XD can be integrated in the core quite easily with only minor modifications.
Best regards,
Maxim Levitsky
Re: Plan for adding XD support in mtd layer
Hello Maxim
Quoted 3 lines · Maxim Levitsky
Currently jmicron support is found in aspire one.
My acer has ricoh device I know how works.
Those are card readers built into notebooks, right? How are they attached? Via PCI or USB?
Quoted 12 lines
Other that work of Alex Dubov, I need to write the code, but I don't want to repeat his mistake and write code that isn't accepted on basis of lack for fitness in existing subsystems.
Quoted 7 lines
Quoted 3 lines
-> I need new
mtd_infomember for at least for card identification, and will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user spaceBlock and page size already exist as
mtd->erasesizeandmtd->writesizerespectively. I'm not sure whether the zone size should become part of the mtd interface - it looks more like a property of the ftl.
Quoted 7 lines
You didn't understand me.
I can't nether write nor erase individual blocks (writing is technically possible, but is disallowed by recent XD spec (this is what Alex Dubov says)However I can _read_ individual sectors
sector is 512 bytes, block is usually around 32 sectors.
It has been a while since I looked at the spec, but I believe there are two parts to it. The flash hardware allows block erases, sector reads and sector writes. On top of that you have the FTL that adds further restrictions.
If you use you xD card without the FTL, as a raw MTD device, you can do almost anything you want. Format it with JFFS2, for example. But if used _with_ the FTL, that card now has a corrupt format and may be unusable.
Or is the FTL already implemented in the card reader hardware?
Quoted 6 lines
Quoted 3 lines
What userspace programs would need this information? I can imagine some sort of
xd_formator mkxd in case the card has been corrupted or used raw. Anything else?I can imagine a format application, a rescue application, and an debug application that will show how healthy is the card.
Format and rescue certainly make sense. I'm not sure how a debug application would measure the health, but maybe that's possible as well.
Quoted 3 lines
There are actually two pieces of the data.
One is an ID, a serial number that can be used for example by udev for permanent mapping.
Ok, makes sense.
Quoted 2 lines
Other is the media type, and this has to be used to know the sizes of sectors, zones, pages, how many spares there are etc...
That would be available when using mtdchar or mtdblock to read the device. Which ought to be enough for format, rescue and debug, I believe.
Quoted 4 lines
ECC corecting will be done in the base mtd driver, but userspace application will be able to determine if block is good or bad. If it is good, it can use oob information to determine of ECC correction was applied or not, and report overall health of the card.
With mtdchar it is possible to do ECC correction in userspace as well. Not that I would consider it a good idea, but the information is certainly available and can be controlled if a compelling reason can be found.
Jörn
Signature · 2 lines
Invincibility is in oneself, vulnerability is in the opponent. -- Sun Tzu
Re: Plan for adding XD support in mtd layer
Quoted 9 lines · Jörn Engel
Hello Maxim
Quoted 3 lines · Maxim Levitsky
Currently jmicron support is found in aspire one.
My acer has ricoh device I know how works.Those are card readers built into notebooks, right? How are they attached? Via PCI or USB?
Mostly in notebooks.
External readers usually implement standard mass-storage protocol.
They are usually PCI , however some USB devices work the same, for example
alluda device that currently has two drivers in the kernel, one mtd for raw access, and another self-contained USB driver that does FTL and exposes device as an SCSI device.
Quoted 18 lines
Quoted 18 lines
Other that work of Alex Dubov, I need to write the code, but I don't want to repeat his mistake and write code that isn't accepted on basis of lack for fitness in existing subsystems.
Quoted 7 lines
Quoted 3 lines
-> I need new
mtd_infomember for at least for card identification, and will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user spaceBlock and page size already exist as
mtd->erasesizeandmtd->writesizerespectively. I'm not sure whether the zone size should become part of the mtd interface - it looks more like a property of the ftl.You didn't understand me.
I can't nether write nor erase individual blocks (writing is technically possible, but is disallowed by recent XD spec (this is what Alex Dubov says)
Quoted 7 lines
Quoted 2 lines
However I can _read_ individual sectors
sector is 512 bytes, block is usually around 32 sectors.It has been a while since I looked at the spec, but I believe there are two parts to it. The flash hardware allows block erases, sector reads and sector writes. On top of that you have the FTL that adds further restrictions.
if this is the case, then no problems.
Alex said something about firmware bugs though, that I belive is connected to raw block writes.
I also know that driver for my ricoh based controller does always block based cow writes.
when I say block I mean area that can be erased independently, and it consists of several 512 byte sectors.
(Very old incarnation of smartmedia format had 256 byte sectors that were tied together in pairs)
Quoted 7 lines
If you use you xD card without the FTL, as a raw MTD device, you can do almost anything you want. Format it with JFFS2, for example. But if used _with_ the FTL, that card now has a corrupt format and may be unusable.
Or is the FTL already implemented in the card reader hardware?
No, I have to implement it.
And yes, XD can give ready to use testing ground for flash based file systems, like JFFS2.
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
What userspace programs would need this information? I can imagine some sort of
xd_formator mkxd in case the card has been corrupted or used raw. Anything else?I can imagine a format application, a rescue application, and an debug application that will show how healthy is the card.
Format and rescue certainly make sense. I'm not sure how a debug application would measure the health, but maybe that's possible as well.
I can count bad sectors, ecc corrected sectors, see how many spares remain in each block, etc...
Quoted 5 lines
Quoted 4 lines
There are actually two pieces of the data.
One is an ID, a serial number that can be used for example by udev for permanent mapping.
Quoted 8 lines
Ok, makes sense.
Quoted 2 lines
Other is the media type, and this has to be used to know the sizes of sectors, zones, pages, how many spares there are etc...
That would be available when using mtdchar or mtdblock to read the device. Which ought to be enough for format, rescue and debug, I believe.
Yes this is enough, but how I expose media type in the mtd driver, there is a field for that I missed?
And same question for the ID.
Note that media type /ID is _not_ stored on the media (well it is but is not accessible that way)
There is need for special command to read it from the device.
Quoted 10 lines
Quoted 5 lines
ECC corecting will be done in the base mtd driver, but userspace application will be able to determine if block is good or bad. If it is good, it can use oob information to determine of ECC correction was applied or not, and report overall health of the card.
With mtdchar it is possible to do ECC correction in userspace as well. Not that I would consider it a good idea, but the information is certainly available and can be controlled if a compelling reason can be found.
Exactly.
Best regards,
Maxim Levitsky
Re: Plan for adding XD support in mtd layer
Quoted 6 lines · Maxim Levitsky
Quoted 3 lines · Jörn Engel
Those are card readers built into notebooks, right? How are they attached? Via PCI or USB?
Mostly in notebooks.
External readers usually implement standard mass-storage protocol.
Ok. I guess I won't buy a new notebook just to test a driver then.
Quoted 5 lines
They are usually PCI , however some USB devices work the same, for example
alluda device that currently has two drivers in the kernel, one mtd for raw access, and another self-contained USB driver that does FTL and exposes device as an SCSI device.
Yes. I've written the mtd driver for that one.
Quoted 10 lines
Quoted 4 lines
It has been a while since I looked at the spec, but I believe there are two parts to it. The flash hardware allows block erases, sector reads and sector writes. On top of that you have the FTL that adds further restrictions.
if this is the case, then no problems.
Alex said something about firmware bugs though, that I belive is connected to raw block writes.I also know that driver for my ricoh based controller does always block based cow writes.
If you only ever want to use the device with the FTL on top, that is perfectly acceptable. But it is an FTL restriction, not part of the hardware. Apart from firmware bugs, of course.
Quoted 4 lines
when I say block I mean area that can be erased independently, and it consists of several 512 byte sectors.
(Very old incarnation of smartmedia format had 256 byte sectors that were tied together in pairs)
Some also had 256 or 512 blocks per zone. These days everything seems to have 512+16 Byte sectors, 16 sectors per block, 1024 physical (1000 logical) blocks per zone and as many zones as the device can fit. Or at least that was the case two years ago.
Quoted 4 lines
Quoted 2 lines
Format and rescue certainly make sense. I'm not sure how a debug application would measure the health, but maybe that's possible as well.
I can count bad sectors, ecc corrected sectors, see how many spares remain in each block, etc...
Iirc the number of bad blocks never changes. Either a block is factory-marked or it will stay in use forever. A smart FTL could try to store the erase count or the number of correctable/uncorrectable errors for each block somewhere, but I don't believe it is done - or could be done while staying compatible with the xD spec.
So that pretty much leaves the number of correctable/uncorrectable errors currently on the device.
Quoted 4 lines
Quoted 3 lines
That would be available when using mtdchar or mtdblock to read the device. Which ought to be enough for format, rescue and debug, I believe.
Quoted 8 lines
Yes this is enough, but how I expose media type in the mtd driver, there is a field for that I missed?
And same question for the ID.
Note that media type /ID is _not_ stored on the media (well it is but is not accessible that way)
There is need for special command to read it from the device.
Right. I forgot about that one. In fact, I tried fairly hard to forget. ;)
According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the first non-defective block. In other words, it can be read and written normally, without any magic. Certainly not the atrocities I tried not to remember.
And even if there are some atrocities hidden, we can abtract them out in the driver. When reading/writing block 0, just do the magic transparently.
Jörn
Signature · 4 lines
There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are no obvious deficiencies. -- C. A. R. Hoare
Re: Plan for adding XD support in mtd layer
Quoted 26 lines · Jörn Engel
Quoted 6 lines · Maxim Levitsky
Quoted 3 lines · Jörn Engel
Those are card readers built into notebooks, right? How are they attached? Via PCI or USB?
Mostly in notebooks.
External readers usually implement standard mass-storage protocol.Ok. I guess I won't buy a new notebook just to test a driver then.
Quoted 5 lines
They are usually PCI , however some USB devices work the same, for example
alluda device that currently has two drivers in the kernel, one mtd for raw access, and another self-contained USB driver that does FTL and exposes device as an SCSI device.Yes. I've written the mtd driver for that one.
Quoted 8 lines
Quoted 4 lines
It has been a while since I looked at the spec, but I believe there are two parts to it. The flash hardware allows block erases, sector reads and sector writes. On top of that you have the FTL that adds further restrictions.
if this is the case, then no problems.
Alex said something about firmware bugs though, that I belive is connected to raw block writes.
Quoted 6 lines
Quoted 2 lines
I also know that driver for my ricoh based controller does always block based cow writes.
If you only ever want to use the device with the FTL on top, that is perfectly acceptable. But it is an FTL restriction, not part of the hardware. Apart from firmware bugs, of course.
Exactly.
Quoted 10 lines
Quoted 5 lines
when I say block I mean area that can be erased independently, and it consists of several 512 byte sectors.
(Very old incarnation of smartmedia format had 256 byte sectors that were tied together in pairs)Some also had 256 or 512 blocks per zone. These days everything seems to have 512+16 Byte sectors, 16 sectors per block, 1024 physical (1000 logical) blocks per zone and as many zones as the device can fit. Or at least that was the case two years ago.
Yes that the same I see.
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
Format and rescue certainly make sense. I'm not sure how a debug application would measure the health, but maybe that's possible as well.
I can count bad sectors, ecc corrected sectors, see how many spares remain in each block, etc...
Iirc the number of bad blocks never changes. Either a block is factory-marked or it will stay in use forever. A smart FTL could try to store the erase count or the number of correctable/uncorrectable errors for each block somewhere, but I don't believe it is done - or could be done while staying compatible with the xD spec.
Quoted 2 lines
So that pretty much leaves the number of correctable/uncorrectable errors currently on the device.
Exactly.
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
That would be available when using mtdchar or mtdblock to read the device. Which ought to be enough for format, rescue and debug, I believe.
Quoted 8 lines
Yes this is enough, but how I expose media type in the mtd driver, there is a field for that I missed?
And same question for the ID.
Note that media type /ID is _not_ stored on the media (well it is but is not accessible that way)
There is need for special command to read it from the device.
Quoted 7 lines
Right. I forgot about that one. In fact, I tried fairly hard to forget. ;)
According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the first non-defective block. In other words, it can be read and written normally, without any magic. Certainly not the atrocities I tried not to remember.
I wish I had the spec.
I don't know the format of the CIS, and I know that special command is used to query the ID.
CIS these days isn't used for anything and I hoped not to touch that thing.
I wont argue on that one with you, but why not to add the new field. It is quite generic, it could be card vendor information or so? Few bits of custom information to assign for each card no?
I am taking about these:
struct xd_card_id1 {
unsigned char maker_code;
unsigned char device_code;
unsigned char option_code1;
unsigned char option_code2;
} __attribute__((packed));
struct xd_card_id2 {
unsigned char characteristics_code;
#define XD_CARD_CELL_TYPE_MASK 0xc0
#define XD_CARD_MULTI_BLOCK_MASK 0x30
unsigned char vendor_code1;
unsigned char size_code;
unsigned char vendor_code2;
unsigned char vendor_code3;
} __attribute__((packed));
struct xd_card_id3 {
unsigned char vendor_code1;
unsigned char vendor_code2;
unsigned char id_code;
unsigned char vendor_code3;
} __attribute__((packed));
Are these 32 bit values copied from CIF?
These values are read using commands:
XD_CARD_CMD_ID1 = 0x90, XD_CARD_CMD_ID2 = 0x91, XD_CARD_CMD_ID3 = 0x9a
I need especially the xd_card_id1.device_code because this specifies the media type, but xd_card_id3.id_code is used to detemine if media is smartmedia or XD.
Best regards,
Maxim Levitsky
Re: Plan for adding XD support in mtd layer
Quoted 6 lines · Maxim Levitsky
Quoted 5 lines
According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the first non-defective block. In other words, it can be read and written normally, without any magic. Certainly not the atrocities I tried not to remember.
I wish I had the spec.
It used to be semi-free. You could download it for your name and address. But I believe I'm not legally able to forward it to you and the official site seems to be down. Most of the information can also be found here:
http://www.win.tue.nl/~aeb/linux/smartmedia/SmartMedia_Format.pdf
And there seems to be a copy available at pudn:
http://en.pudn.com/downloads137/doc/fileformat/detail585996_en.html
Quoted 4 lines
I don't know the format of the CIS, and I know that special command is used to query the ID.
CIS these days isn't used for anything and I hoped not to touch that thing.
:)
Quoted 37 lines
I wont argue on that one with you, but why not to add the new field. It is quite generic, it could be card vendor information or so? Few bits of custom information to assign for each card no?
I am taking about these:
struct
xd_card_id1{
unsigned charmaker_code;
unsigned chardevice_code;
unsigned charoption_code1;
unsigned charoption_code2;
} __attribute__((packed));struct
xd_card_id2{
unsigned charcharacteristics_code;
#defineXD_CARD_CELL_TYPE_MASK0xc0
#defineXD_CARD_MULTI_BLOCK_MASK0x30unsigned char
vendor_code1;
unsigned charsize_code;
unsigned charvendor_code2;
unsigned charvendor_code3;
} __attribute__((packed));struct
xd_card_id3{
unsigned charvendor_code1;
unsigned charvendor_code2;
unsigned charid_code;
unsigned charvendor_code3;
} __attribute__((packed));Are these 32 bit values copied from CIF?
These values are read using commands:
Quoted 8 lines
XD_CARD_CMD_ID1 = 0x90, XD_CARD_CMD_ID2 = 0x91, XD_CARD_CMD_ID3 = 0x9aI need especially the xd_card_id1.device_code because this specifies the media type, but xd_card_id3.id_code is used to detemine if media is smartmedia or XD.
If you add this information to a struct xD_card and have mtd->priv point to said structure, wouldn't that be enough for your purposes?
Jörn
Signature · 3 lines
The cheapest, fastest and most reliable components of a computer system are those that aren't there. -- Gordon Bell, DEC labratories
Re: Plan for adding XD support in mtd layer
Quoted 6 lines · Jörn Engel
Quoted 5 lines · Maxim Levitsky
Quoted 5 lines
According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the first non-defective block. In other words, it can be read and written normally, without any magic. Certainly not the atrocities I tried not to remember.
Quoted 7 lines
Quoted 1 line
I wish I had the spec.
It used to be semi-free. You could download it for your name and address. But I believe I'm not legally able to forward it to you and the official site seems to be down. Most of the information can also be found here:
http://www.win.tue.nl/~aeb/linux/smartmedia/SmartMedia_Format.pdf
This one I have, and there is a brief description of the CIS format, which seems to indicate that its contents are predefined.
Quoted 52 lines
And there seems to be a copy available at pudn:
http://en.pudn.com/downloads137/doc/fileformat/detail585996_en.htmlQuoted 4 lines
I don't know the format of the CIS, and I know that special command is used to query the ID.
CIS these days isn't used for anything and I hoped not to touch that thing.:)
Quoted 42 lines
I wont argue on that one with you, but why not to add the new field. It is quite generic, it could be card vendor information or so? Few bits of custom information to assign for each card no?
I am taking about these:
struct
xd_card_id1{
unsigned charmaker_code;
unsigned chardevice_code;
unsigned charoption_code1;
unsigned charoption_code2;
} __attribute__((packed));struct
xd_card_id2{
unsigned charcharacteristics_code;
#defineXD_CARD_CELL_TYPE_MASK0xc0
#defineXD_CARD_MULTI_BLOCK_MASK0x30unsigned char
vendor_code1;
unsigned charsize_code;
unsigned charvendor_code2;
unsigned charvendor_code3;
} __attribute__((packed));struct
xd_card_id3{
unsigned charvendor_code1;
unsigned charvendor_code2;
unsigned charid_code;
unsigned charvendor_code3;
} __attribute__((packed));Are these 32 bit values copied from CIF?
These values are read using commands:
XD_CARD_CMD_ID1 = 0x90, XD_CARD_CMD_ID2 = 0x91, XD_CARD_CMD_ID3 = 0x9a
Quoted 6 lines
Quoted 3 lines
I need especially the xd_card_id1.device_code because this specifies the media type, but xd_card_id3.id_code is used to detemine if media is smartmedia or XD.
If you add this information to a struct
xD_cardand havemtd->privpoint to said structure, wouldn't that be enough for your purposes?
I am afraid not.
My plan is to create one 'meta' mtd driver for all XD controllers, and plug in small drivers.
The meta driver and small driver will share the ->priv structure.
I need a way to pass the ID from mtd driver to FTL, and to userspace via mtdchar.
Best regards,
Maxim Levitsky
Re: Plan for adding XD support in mtd layer
Quoted 3 lines · Maxim Levitsky
This one I have, and there is a brief description of the CIS format, which seems to indicate that its contents are predefined.
That is my understanding, yes. Essentially an xD card is just nand flash in cheap packaging with the first block filled by CIS/IDI.
Quoted 10 lines
I am afraid not.
My plan is to create one 'meta' mtd driver for all XD controllers, and plug in small drivers.
The meta driver and small driver will share the ->priv structure.
I need a way to pass the ID from mtd driver to FTL, and to userspace via mtdchar.
You can also add a ->priv field to struct xD_data. Or you can have it contain the mtd_info, and use container_of to access the xD information.
But those are details. Feel free to write the code either way. We can sort out which method to use later.
Jörn
Signature · 2 lines
Audacity augments courage; hesitation, fear. -- Publilius Syrus
Re: Plan for adding XD support in mtd layer
Quoted 16 lines · Jörn Engel
Quoted 3 lines · Maxim Levitsky
This one I have, and there is a brief description of the CIS format, which seems to indicate that its contents are predefined.
That is my understanding, yes. Essentially an xD card is just nand flash in cheap packaging with the first block filled by CIS/IDI.
Quoted 8 lines
I am afraid not.
My plan is to create one 'meta' mtd driver for all XD controllers, and plug in small drivers.
The meta driver and small driver will share the ->priv structure.
Quoted 8 lines
Quoted 2 lines
I need a way to pass the ID from mtd driver to FTL, and to userspace via mtdchar.
You can also add a ->priv field to struct
xD_data. Or you can have it contain themtd_info, and usecontainer_ofto access the xD information.But those are details. Feel free to write the code either way. We can sort out which method to use later.
Thanks, no more questions, I will publish code as soon as I write one.
Best regards,
Maxim Levitsky
Quoted 3 lines
Jörn
Quoted 5 lines
The "mtd uses blocking calls" argument I can help
with. For some other
project I've added non-blocking calls to mtd[1].
Quoted 8 lines
Biggest problems I see are a) the userspace interface and
b) allowing
both raw flash access through MTD and block device access with an FTL.[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I actually tried several approaches to this problem and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
Recently I was going to go back to it, but then I was asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.
Quoted 10 lines · Alex Dubov
Quoted 10 lines
The "mtd uses blocking calls" argument I can help
with. For some other
project I've added non-blocking calls to mtd[1].Biggest problems I see are a) the userspace interface and
b) allowing
both raw flash access through MTD and block device access with an FTL.
Quoted 7 lines
Quoted 3 lines
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I actually tried several approaches to this problem and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
Recently I was going to go back to it, but then I was asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.
Hi,
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Also, as I understand the command interface more and more, it seems that 'magically' xD card had same interface as standard NAND flash chip.
I think I can implement the driver for each controller just like an nand chip driver.
Best regards,
Maxim Levitsky
Quoted 30 lines · Maxim Levitsky
From: Maxim Levitsky <[email protected]>
Subject: Re: XD/smartmedia - how to implement it right?
To: "Alex Dubov" <[email protected]>
Cc: "Jörn Engel" <[email protected]>, "linux-kernel" <[email protected]>, [email protected], [email protected]
Date: Saturday, November 28, 2009, 2:36 AMQuoted 2 lines · Alex Dubov
Quoted 2 lines
The "mtd uses blocking calls" argument I can
help
Quoted 4 lines
Quoted 4 lines
with. For some other
project I've added non-blocking calls to mtd[1].Biggest problems I see are a) the userspace
interface and
Quoted 2 lines
Quoted 2 lines
b) allowing
both raw flash access through MTD and blockdevice access
Quoted 7 lines
Quoted 5 lines
with an FTL.
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I actually tried several approaches to this problem
and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Recently I was going to go back to it, but then I was
asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.
Hi,
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Because it's the future. ;-)
Quoted 10 lines
Also, as I understand the command interface more and more, it seems that
'magically' xD card had same interface as standard NAND flash chip.I think I can implement the driver for each controller just like an nand
chip driver.
xD spec, first of all, is about flash-based ordinary block devices. All the hard problems are in good r/w FTL implementation, which is currently mostly absent from the MTD.
Quoted 37 lines · Alex Dubov
Quoted 34 lines · Maxim Levitsky
From: Maxim Levitsky <[email protected]>
Subject: Re: XD/smartmedia - how to implement it right?
To: "Alex Dubov" <[email protected]>
Cc: "Jörn Engel" <[email protected]>, "linux-kernel" <[email protected]>, [email protected], [email protected]
Date: Saturday, November 28, 2009, 2:36 AMQuoted 2 lines · Alex Dubov
Quoted 2 lines
The "mtd uses blocking calls" argument I can
help
Quoted 4 lines
Quoted 4 lines
with. For some other
project I've added non-blocking calls to mtd[1].Biggest problems I see are a) the userspace
interface and
Quoted 2 lines
Quoted 2 lines
b) allowing
both raw flash access through MTD and blockdevice access
Quoted 7 lines
Quoted 5 lines
with an FTL.
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I actually tried several approaches to this problem
and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
Quoted 2 lines
Recently I was going to go back to it, but then I was
asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.
Quoted 6 lines
Quoted 4 lines
Hi,
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Because it's the future. ;-)
Although I like the xD format, it has no future,
Everyone migrated to SD now.
Reasons are very simple:
1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many readers couldn't use it.
I know that windows driver I was dealing with supports sizes up to 2GB
2 - xD is a nand card, but with 8-bit bus.
Newer nand chips seem to favor 16 bit bus or more. Obviously, if they were to implement that it would be physically incompatible.
3 - embedded cpus are cheap dirt now, and allow to use their, always 'better' FTL strategy, and also hide inner working of the chip.
Memstick pro is also today sort of SD card, but for Sony its their first priority to keep proprietary standards even if inferior.
I won't be surprised though if this format dies too.
Could you explain why we need an asynchronous interface? I understand that filesystem/page cache/block cache/and block layer, already provide caches and asynchronous behavior.
Only maybe reading several pages at time, and writing could benefit, but many devices don't support that feature.
And again, bulk of the time will be spend writing, and only saving will be from queuing more work to the device.
Quoted 7 lines
Quoted 7 lines
Also, as I understand the command interface more and more, it seems that
'magically' xD card had same interface as standard NAND flash chip.
Quoted 8 lines
Quoted 4 lines
I think I can implement the driver for each controller just like an nand
chip driver.xD spec, first of all, is about flash-based ordinary block devices. All the hard problems are in good r/w FTL implementation, which is currently mostly absent from the MTD.
Its true, and I will write an r/w FTL for xD devices, in same manner as existing FTL implementations.
It works above the mtd layer btw.
Best regards,
Maxim Levitsky
Quoted 89 lines · Maxim Levitsky
Quoted 37 lines · Alex Dubov
Quoted 34 lines · Maxim Levitsky
From: Maxim Levitsky <[email protected]>
Subject: Re: XD/smartmedia - how to implement it right?
To: "Alex Dubov" <[email protected]>
Cc: "Jörn Engel" <[email protected]>, "linux-kernel" <[email protected]>, [email protected], [email protected]
Date: Saturday, November 28, 2009, 2:36 AMQuoted 2 lines · Alex Dubov
Quoted 2 lines
The "mtd uses blocking calls" argument I can
help
Quoted 4 lines
Quoted 4 lines
with. For some other
project I've added non-blocking calls to mtd[1].Biggest problems I see are a) the userspace
interface and
Quoted 2 lines
Quoted 2 lines
b) allowing
both raw flash access through MTD and blockdevice access
Quoted 7 lines
Quoted 5 lines
with an FTL.
[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
Jörn
I actually tried several approaches to this problem
and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
Quoted 2 lines
Recently I was going to go back to it, but then I was
asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.
Quoted 6 lines
Quoted 4 lines
Hi,
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Because it's the future. ;-)
Although I like the xD format, it has no future,
Everyone migrated to SD now.Reasons are very simple:
1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many readers couldn't use it.
I know that windows driver I was dealing with supports sizes up to 2GB2 - xD is a nand card, but with 8-bit bus.
Newer nand chips seem to favor 16 bit bus or more. Obviously, if they were to implement that it would be physically incompatible.3 - embedded cpus are cheap dirt now, and allow to use their, always 'better' FTL strategy, and also hide inner working of the chip.
Memstick pro is also today sort of SD card, but for Sony its their first priority to keep proprietary standards even if inferior.
I won't be surprised though if this format dies too.Could you explain why we need an asynchronous interface? I understand that filesystem/page cache/block cache/and block layer, already provide caches and asynchronous behavior.
Only maybe reading several pages at time, and writing could benefit, but many devices don't support that feature.
And again, bulk of the time will be spend writing, and only saving will be from queuing more work to the device.Quoted 12 lines
Quoted 11 lines
Also, as I understand the command interface more and more, it seems that
'magically' xD card had same interface as standard NAND flash chip.I think I can implement the driver for each controller just like an nand
chip driver.
Quoted 7 lines
Quoted 3 lines
xD spec, first of all, is about flash-based ordinary block devices. All the hard problems are in good r/w FTL implementation, which is currently mostly absent from the MTD.
Its true, and I will write an r/w FTL for xD devices, in same manner as existing FTL implementations.
It works above the mtd layer btw.
Alex, could you do me a favor, and explain why partial block writes don't work?
What happens if its done?
the FTL layer of course will do full block writes, but its not easy to do full block writes in mtd layer.
If I was going to do full block writes, I would need to set page_size = block size, and then loose many features nand mtd supports.
Best regards,
Maxim Levitsky
Quoted 4 lines · Maxim Levitsky
Alex, could you do me a favor, and explain why partial block writes don't work?
What happens if its done?
With the Alauda and the limited set of cards I tried they work.
Jörn
Signature · 4 lines
The story so far: In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move. -- Douglas Adams
Quoted 9 lines · Jörn Engel
From: Jörn Engel <[email protected]>
Subject: Re: XD/smartmedia - how to implement it right?
To: "Maxim Levitsky" <[email protected]>
Cc: "Alex Dubov" <[email protected]>, "linux-kernel" <[email protected]>, [email protected], [email protected]
Date: Tuesday, December 1, 2009, 12:22 AMQuoted 2 lines · Maxim Levitsky
Alex, could you do me a favor, and explain why partial
Quoted 9 lines
block writes
Quoted 2 lines
don't work?
What happens if its done?With the Alauda and the limited set of cards I tried they work.
Jörn
It's a bug particular to jmicron controllers.
Quoted 16 lines · Alex Dubov
Quoted 13 lines · Jörn Engel
From: Jörn Engel <[email protected]>
Subject: Re: XD/smartmedia - how to implement it right?
To: "Maxim Levitsky" <[email protected]>
Cc: "Alex Dubov" <[email protected]>, "linux-kernel" <[email protected]>, [email protected], [email protected]
Date: Tuesday, December 1, 2009, 12:22 AMQuoted 2 lines · Maxim Levitsky
Alex, could you do me a favor, and explain why partial
block writes
Quoted 2 lines
don't work?
What happens if its done?
Quoted 7 lines
Quoted 5 lines
With the Alauda and the limited set of cards I tried they work.
Jörn
It's a bug particular to jmicron controllers.
Thanks!
So, I will write the driver for my controller using nand subsystem, and then, I see what to do next.
If you write your mtd driver, I have no objection to port my driver to yours (I assume that there will be a common core, just like the nand subsystem, and small drivers.)
And the last question probably:
I know that xD is very similar to nand, but due to things like the above, maybe full mtd driver is really better.
There will be some code duplication, but on the other hand the driver could follow the way windows driver works more closely, thus making it more compatable with xD cards.
So if I or Alex write the full blown mtd driver, there will be no objections because of code duplication?
Anyway, now I am going to stop talking and start coding (I did quite in depth review of mtd and nand systems already)
Best regards,
Maxim Levitsky
Quoted 10 lines · Jörn Engel
Quoted 4 lines · Maxim Levitsky
Alex, could you do me a favor, and explain why partial block writes don't work?
What happens if its done?With the Alauda and the limited set of cards I tried they work.
Jörn
I have few severe problems in writing the driver, but otherwise its is almost written. I also suspect that I will have to modify mtd core in several places to achieve what I need to. I guess Alex was right after all...
First of all, the device I deal with supports 32 bit dma only, as well as many other pci devices. (I know that for sure because dma register is 32 bit wide....)
Now, mtd_blkdevs doesn't ask block subsystem for 32 bit bounce, and in addition to that there is no guarantee that my mtd driver won't be called on unrelated kernel buffer (for example if ftl layer does caching....)
So I guess, ether I need to try to map the buffer, and then fail back to copying into preallocated buffer in 32 bit zone, or always use that buffer.
MMC core seem to handle the dma bouncing inside.
But that minor problem, other problem, I revealed when I began to think about how properly to guard against races.
Problem is that mtd core doesn't handle hot-plug well.
Suppose, user has mounted filesystem on xd card and pulls it out. SD/MMC cards handle that case. There might be fs corruption though, but that is the user responsibility.
Now if anything has open the mtd_blkdevs, it increments the mtd user count, thus removal of the mtd device will just fail.
Now I really don't know how to handle that property, but I know that yanking out usb keys, mmc cards, etc works, and all devices that were created disappear.
I will eventually figure out how to do that right, but it would be great if you help me a bit.
Best regards,
Maxim Levitsly
Quoted 15 lines
Quoted 5 lines
Quoted 3 lines
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Because it's the future. ;-)
Although I like the xD format, it has no future,
Everyone migrated to SD now.Reasons are very simple:
1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many readers couldn't use it.
I know that windows driver I was dealing with supports sizes up to 2GB
Quoted 7 lines
2 - xD is a nand card, but with 8-bit bus.
Newer nand chips seem to favor 16 bit bus or more. Obviously, if they were to implement that it would be physically incompatible.3 - embedded cpus are cheap dirt now, and allow to use their, always 'better' FTL strategy, and also hide inner working of the chip.
XDs are still used by at least olympus digital cameras. Ouch.
If you have onboard embedded cpu, you can bet it will not work correctly over powerfail. If you do FTL on host, you have at least chance to get it right.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Quoted 19 lines
Quoted 4 lines
Quoted 2 lines
First of all, thank you very much for your contributions. Could you explain, why we need an asynchronous mtd driver?
Because it's the future. ;-)
Although I like the xD format, it has no future,
Everyone migrated to SD now.Reasons are very simple:
1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many readers couldn't use it.
I know that windows driver I was dealing with supports sizes up to 2GB2 - xD is a nand card, but with 8-bit bus.
Newer nand chips seem to favor 16 bit bus or more. Obviously, if they were to implement that it would be physically incompatible.
Quoted 4 lines
Quoted 2 lines
3 - embedded cpus are cheap dirt now, and allow to use their, always 'better' FTL strategy, and also hide inner working of the chip.
XDs are still used by at least olympus digital cameras. Ouch.
Well, at least according to wikipedia, Olympus phased out XD support, as well as Fuji.
This doesn't mean that there will be no users of XD in five following years, thats why I am writing the driver in first place.
Quoted 4 lines
If you have onboard embedded cpu, you can bet it will not work correctly over powerfail. If you do FTL on host, you have at least chance to get it right.
Thats correct. In fact we are both right here.
Your viewpoint is based on a view of an user of a sane operation system. Sure, the whole idea of FTL is evil, the perfect place for wear leveling is in the filesystem itself, and in addition to that, there are several technologies that imply finite write cycle count, and that reason justifies such filesystem even more.
I also use the same sane operation system, so for me its quite nice to have XD^WNAND removable chip support, and in my eyes the raw access to flash is very nice advantage.
However, due to limitations of other less sane operation systems, hardware makers have to support wear leveling ether in the driver or in the hardware.
This is yet another example, about why some operation systems 'just work'...
Not that these operation systems are better, but that hardware makers do everything possible to patch all problems these operation systems have. Same is true for software generally.
Supporting wear leveling in hardware is easier, because this way, they can be sure it works well, and not depend on each and every driver implementation to do that right.
Small hint, you know how the windows driver for my controller handles COW?
Very simple, every 512 bytes write (only accepts such) turns into whole erase/write cycle of whole block (32 KB usually).
So, if you write a 32K or larger file , and no matter how clever the OS is, it will still end in 64 erase cycles...
Best regards,
Maxim Levitsky
Quoted 3 lines · Maxim Levitsky
Supporting wear leveling in hardware is easier, because this way, they can be sure it works well, and not depend on each and every driver implementation to do that right.
Assuming that hardware engineers know what they are doing, yes. Unfortunately the hardware people seem to be just as clueless about this as the every device driver writer, which actually makes this much worse. Having bad wear leveling in hardware often means that the card eats your data, no matter what the driver does. Doing it in software means that it's broken most of the time but fixable.
Arnd <><